Eye center of rotation determination, depth plane selection, and render camera positioning in display systems

ABSTRACT

A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system images the user&#39;s eye and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of rotation of the user&#39;s eye. The display system may render virtual image content with a render camera positioned at the determined position of the center of rotation of said eye.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/250,931, which is titled “EYE CENTER OF ROTATIONDETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING INDISPLAY SYSTEMS” and was filed on Jan. 17, 2019; which claims priorityto U.S. Patent Prov. App. 62/618,559, which is titled “EYE CENTER OFROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERAPOSITIONING IN DISPLAY SYSTEMS” and was filed on Jan. 17, 2018. Thisapplication further claims priority to U.S. Patent Prov. App.62/702,849, which is titled “EYE CENTER OF ROTATION DETERMINATION, DEPTHPLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS” andwas filed on Jul. 24, 2018. Each of the above-recited applications isincorporated herein by reference in its entirety.

This application further incorporates by reference U.S. Patent Pub.2018/0018515, which is titled “IRIS BOUNDARY ESTIMATION USING CORNEACURVATURE” and was published on Jan. 18, 2018.

FIELD

The present disclosure relates to display systems, virtual reality, andaugmented reality imaging and visualization systems and, moreparticularly, to depth plane selection based in part on a user'sinterpupillary distance.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Various examples of depth plane selection in a mixed reality system aredisclosed.

A display system can be configured to project light to an eye of a userto display virtual image content in a vision field of said user. Theuser's eye may have a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea. The displaysystem can include a frame configured to be supported on a head of theuser, a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field at different amounts of atleast one of divergence and collimation and thus the displayed virtualimage content appears to originate from different depths at differentperiods of time, one or more eye tracking cameras configured to imagethe user's eye, and processing electronics in communication with thedisplay and the one or more eye tracking cameras, the processingelectronics configured to obtain an estimate of a center of rotation ofsaid eye based on images of said eye obtained with said one or more eyetracking cameras.

Various examples of display systems that project light to one or moreeyes of a user to display virtual image content in a vision field ofsaid user are described herein such as the examples enumerated below:

Example 1: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field at different amounts of at least one ofdivergence and collimation and thus the displayed virtual image contentappears to originate from different depths at different periods of time;

one or more eye tracking cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore eye tracking cameras, the processing electronics configured toobtain an estimate of a center of rotation of said eye based on imagesof said eye obtained with said one or more eye tracking cameras.

Example 2: The display system of Example 1, further comprising one ormore light sources disposed on said frame with respect to said user'seye to illuminate said user's eye, said one or more eye tracking camerasforming images of said eye using said light from said one or more lightsources.

Example 3: The display system of Example 1 or 2, wherein said one ormore light sources comprises at least two light sources disposed on saidframe with respect to said user's eye to illuminate said user's eye.

Example 4: The display system of Example 1 or 3, wherein said one ormore light sources comprises infrared light emitters.

Example 5: The display system of any of the Examples 1 to 4, wherein oneor more light sources form one or more glints on said eye and saidprocessing electronics is configured to determine a location of saidcornea based on said one or more glints.

Example 6: The display system of any of the Examples 1 to 5, whereinsaid cornea has associated therewith a cornea sphere having a center ofcurvature and said processing electronics is configured to determine alocation of said center of curvature of said cornea sphere.

Example 7: The display system of Example 5, wherein said cornea hasassociated therewith a cornea sphere having a center of curvature andsaid processing electronics is configured to determine a location ofsaid center of curvature of said cornea sphere based on said one or moreglints.

Example 8: The display system of any of the Examples above, wherein saidone or more eye tracking camera is configured to image said pupil ofsaid eye.

Example 9: The display system of any of the Examples above, wherein saidprocessing electronics is configured to determine the location of saidcenter of said pupil.

Example 10: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine at least aportion of a boundary between said iris and said pupil.

Example 11: The display system of Example 10, wherein said processingelectronics is configured to determine a center of said boundary betweensaid iris and said pupil.

Example 12: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of said pupil in three-dimensional space with respect to acenter of curvature of said cornea.

Example 13: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of said optical axis.

Example 14: The display system of Example 12, wherein said processingelectronics is configured to determine a location and orientation ofsaid optical axis based on a location of said center of said pupil inthree-dimensional space.

Example 15: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine said location andorientation of said optical axis based on a location of said center ofsaid pupil in three-dimensional space with respect to a center ofcurvature of said cornea.

Example 16: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea.

Example 17: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea and a location and orientation of said optical axis.

Example 18: The display system of Example 17, wherein said processingelectronics is configured to determine the location of said center ofrotation of said eye by translating a particular distance along saidoptical axis from said center of curvature of said cornea.

Example 19: The display system of Example 18, wherein said particulardistance from said center of curvature to said center of rotation isbetween 4.0 mm and 6.0 mm

Example 20: The display system of Example 18 or 19, wherein saidparticular distance from said center of curvature to said center ofrotation is about 4.7 mm.

Example 21: The display system of Example 18 or 19, wherein saidparticular distance is fixed.

Example 22: The display system of Example 18 or 19, wherein saidprocessing electronics is configured to determine the particulardistance based at least on one or more images of said eye previouslyobtained with said one or more eye tracking cameras.

Example 23: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis, offset from said optical axis, based onsaid location and orientation of said optical axis.

Example 24: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis based on an angular rotation with respectto said optical axis.

Example 25: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis based on an angular rotation of between4.0° and 6.5° with respect to said optical axis.

Example 26: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis based on an angular rotation of about 5.2°with respect to said optical axis.

Example 27: The display system of any of the Examples above, whereinsaid processing electronics are configured to determine a location andorientation of a visual axis based at least on one or more images ofsaid eye previously obtained with said one or more eye tracking cameras.

Example 28: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a center ofrotation of said eye based multiple determinations of said location ofsaid optical axis or visual axis over a period of time during which saideye is rotating.

Example 29: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine said center ofrotation by identifying a region of intersection, convergence, or closeproximity of multiple determinations of said location of said opticalaxis or a visual axis over a period of time during which said eye isrotating.

Example 30: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance of said user where left and right eyes of a user are gazingbased on a determination of the location and orientation of said opticalaxes for said left and right eyes of the user.

Example 31: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance of said user where left and right eyes of a user are gazingbased on a determination of the location and orientation of said visualaxes for said left and right eyes of the user.

Example 32: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance where left and right eyes of a user are gazing based onidentifying a region of intersection, convergence, or close proximity ofsaid visual axes for said left and right eyes of the user.

Example 33: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance where left and right eyes of a user are gazing by projectingthe visual axes for said left and right eyes onto a horizontal plane andidentifying a region of intersection, convergence, or close proximity ofsaid projections of the visual axes for said left and right eyes onto ahorizontal plane.

Example 34: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the relativeamounts of at least one of divergence, and collimation to project imagecontent based on a determination of said vergence distance.

Example 35: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 36: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 37: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 38: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 39: The display system of any of the Examples above, whereinsaid head-mounted display receives light from a portion of theenvironment in front of the user at a first amount of divergence andtransmits the light from the portion of the environment in front of theuser to the user's eye with a second amount of divergence that issubstantially similar to the first amount of divergence.

Example 40: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain the estimate of thecenter of rotation by filtering, averaging, applying a Kalman filter, orany combinations thereof a plurality of estimated center of rotationpositions.

Example 41: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that is rendered as if captured by a camerahaving an aperture at the determined position of the center of rotationof said user's eye.

Example 42: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera at saidcenter of rotation to render virtual images to be presented to said eye.

Example 43: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render cameraconfigured to render virtual images to be presented to said eye that arerendered as if captured by a camera having an aperture closer to saidthe center of rotation than said retina of said eye.

Example 44: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render cameraconfigured to render virtual images to be presented to said eye that arerendered as if captured by a camera having an aperture at said thecenter of rotation of said eye.

Example 45: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera at saidcenter of rotation to render virtual images to be presented to said eye,said render camera modeled with an aperture at said center of rotationof said eye.

Example 46: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field at different amounts of at least one ofdivergence and collimation and thus the displayed virtual image contentappears to originate from different depths at different periods of time;

one or more eye tracking cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore eye tracking cameras, the processing electronics configured toobtain a position estimate of a center of perspective of said eye basedon images of said eye obtained with said one or more eye trackingcameras, said center of perspective being estimated to be proximal saidpupil of said eye or between said cornea and said pupil of said eye,

wherein said processing electronics is configured to present saidvirtual image content to said user's eye that are rendered by a rendercamera located at said center of perspective.

Example 47: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than saidretina.

Example 48: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than a center ofrotation of the eye.

Example 49: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture at said center of perspective.

Example 50: The display system of any of the Examples above, whereinsaid center of perspective is not located at said pupil of said eye.

Example 51: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain an estimate of saiduser's eye pose over time and wherein the processing electronics adjustthe position of the render camera based at least in part upon the user'seye pose.

Example 52: The display system of any of the Examples above, wherein theprocessing electronics is configured to track said user's eye pose overtime and wherein the position of the render camera is adjusted over timein response to changes in said user's eye pose over time.

Example 53: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain the estimate of thecenter of perspective by filtering a plurality of estimated center ofperspective positions.

Example 54: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain the estimate of thecenter of perspective by averaging and/or applying a Kalman filter to aplurality of estimated center of perspective positions.

Example 55: The display system of any of the Examples above, wherein thecenter of perspective comprises a position within the anterior chamberof said user's eye.

Example 56: The display system of any of the Examples above, wherein thecenter of perspective comprises a position in front of said pupil ofsaid user's eye.

Example 57: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 1.0 mm and2.0 mm in front of said pupil of said user's eye.

Example 58: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is about 1.0 mm in frontof said pupil of said user's eye.

Example 59: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 0.25 mm and1.0 mm in front of said pupil of said user's eye.

Example 60: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 0.5 mm and1.0 mm in front of said pupil of said user's eye.

Example 61: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 0.25 mm and0.5 mm in front of said pupil of said user's eye.

Example 62: The display system of any of the Examples above, wherein thecenter of perspective lies along the optical axis of said eye andwherein said processing electronics are further configured to obtain theposition estimate of the center of perspective by obtaining a positionestimate of the optical axis of said eye.

Example 63: The display system of any of the Examples above, wherein thecenter of perspective lies along the optical axis of said eye at aposition between an outer surface of the cornea and the pupil of saideye and wherein said processing electronics are further configured toobtain the position estimate of the center of perspective by obtaining aposition estimate of the optical axis of said eye.

Example 64: The display system of any of the Examples above, wherein thecenter of perspective lies along the optical axis of said eye at aposition between an outer surface of the cornea and the pupil of saideye and wherein said processing electronics are further configured toobtain the position estimate of the center of perspective by obtaining aposition estimate of the optical axis of said eye and a positionestimate of a center of rotation of said eye, the cornea of said eye,the iris of said eye, the retina of said eye, and the pupil of said eyeor any combinations thereof.

Example 65: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 66: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 67: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 68: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 69: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or more eyetracking cameras capturing images of said eye using said light from saidone or more light sources.

Example 70: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 71: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least three light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 72: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 73: The display system of any of the Examples above, whereinsaid one or more light sources form one or more glints on said eye andsaid processing electronics is configured to determine the position ofthe center of curvature of said cornea based on said one or more glints.

Example 74: The display system of any of the Examples above, whereinsaid one or more light sources form one or more glints on said eye andsaid processing electronics is configured to determine athree-dimensional position of the center of curvature of said corneabased on said one or more glints.

Example 75: The display system of any of the Examples above, whereinsaid one or more eye-tracking cameras are further configured to imagesaid pupil of the user's eye and wherein said processing electronics arefurther configured to determine the position of said pupil of said eyebased at least on the image of said pupil from the one or moreeye-tracking cameras.

Example 76: The display system of any of the Examples above, whereinsaid one or more eye-tracking cameras are further configured to imagesaid pupil of the user's eye and wherein said processing electronics arefurther configured to determine a three-dimensional position of saidpupil of said eye based at least on the image of said pupil from the oneor more eye-tracking cameras.

Example 77: The display system of any of the Examples above, whereinsaid one or more eye-tracking cameras are further configured to imagesaid pupil of the user's eye and wherein said processing electronics arefurther configured to determine the position of said pupil of said eyebased on the position of the center of curvature of said cornea andbased on the image of said pupil from the one or more eye-trackingcameras.

Example 78: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the optical axisof said eye based on the three-dimensional position of the center ofcurvature of said cornea and based on the three-dimensional position ofsaid pupil.

Example 79: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a visual axis ofsaid eye based on the optical axis.

Example 80: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the visual axisof said eye based on the optical axis and the three-dimensional positionof at least one of the center of curvature of said cornea, said pupil orboth.

Example 81: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine athree-dimensional position of the center of rotation of said eye basedon the three-dimensional position of the center of curvature of saidcornea.

Example 82: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine athree-dimensional position of the center of rotation of said eye basedon the three-dimensional position of the center of curvature of saidcornea and based on said optical axis.

Example 83: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the distancebetween said eye and the additional eye of said user based at least onthe three-dimensional position of the center of rotation of said eye.

Example 84: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine an interpupillarydistance between said eye and the additional eye of said user based atleast on the three-dimensional position of the center of rotation ofsaid eye.

Example 85: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the optical axis of said eye.

Example 86: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the optical axis of said eye andon a determined optical axis of the additional eye of said user.

Example 87: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the visual axis of said eye andon a determined visual axis of the additional eye of said user.

Example 88: The display system of any of the Examples above, whereinsaid display is configured to project collimated light into said user'seye.

Example 89: The display system of any of the Examples above, whereinsaid display is configured to project collimated light corresponding toan image pixel into said user's eye at a first period of time anddivergent light corresponding to said image pixel into said user's eyeat a second period of time.

Example 90: The display system of any of the Examples above, whereinsaid display is configured to project light corresponding to an imagepixel having a first amount of divergence into said user's eye at afirst period of time and to project light corresponding to said imagepixel having a second amount of divergence, greater than the firstamount of divergence, into said user's eye at a second period of time.

Example 91: A method of rendering virtual image content in a displaysystem configured to project light to an eye of a user to display thevirtual image content in a vision field of said user, said eye having acornea, an iris, a pupil, a lens, a retina, and an optical axisextending through said lens, pupil, and cornea, said method comprising:

with one or more eye tracking cameras configured to image said eye ofthe user to track movements of said eye, determining a position of acenter of rotation of said eye;

with a render engine, rendering virtual image content with a rendercamera at said center of rotation of said eye, said render cameraconfigured to render virtual images to be presented to said eye; and

with a head-mounted display, projecting light into said user's eye todisplay the rendered virtual image content to the user's vision field atdifferent amounts of divergence such that the virtual image contentappears to originate from different depths at different periods of time.

Example 92: The method of any of the Examples above, wherein said rendercamera is configured to render virtual images to be presented to saideye that are rendered as if captured by a camera having an aperturecloser to said the center of rotation than said retina of said eye.

Example 93: The method of any of the Examples above, wherein said rendercamera is configured to render virtual images to be presented to saideye that are rendered as if captured by a camera having an aperture atsaid the center of rotation.

Example 94: The method of any of the Examples above, wherein said rendercamera is modeled with an aperture at said center of rotation of saideye.

Example 95: The method of any of the Examples above, wherein said rendercamera is modeled with an aperture, a lens, and a detector.

Example 96: The method of any of the Examples above, wherein said rendercamera has an aperture at a position along a line between (i) thedetermined position of the center of rotation of said eye and (ii) thedetermined position of the at least one of said iris or pupil.

Example 97: The method of any of the Examples above, further comprising:

with the one or more eye tracking cameras, determining a position of acenter of perspective of said user's eye, wherein the center ofperspective of said user's eye is located less than approximately 1.0 mmfrom the pupil of said user's eye; and

with the render engine, rendering the virtual image content with therender camera,

wherein said render camera has an aperture at the determined position ofthe center of perspective of said user's eye.

Example 98: The method of any of the Examples above, further comprising:

with the render engine, rendering the virtual image content with therender camera, wherein said render camera has an aperture at a positionalong a line between (i) the determined position of the center ofrotation of said eye and (ii) the determined position of the center ofperspective of said user's eye.

Example 99: The method of any of the Examples above, further comprising:

with processing electronics in communication with the one or more eyetracking cameras, determining a measure of change with time of thedetermined position of the center of perspective of said user's eye; and

with the processing electronics, if it is determined that the measure ofchange with time exceeds a first threshold, directing the render engineto render the virtual content with the render camera, wherein the rendercamera has an aperture at the determined position of the center ofrotation of said eye.

Example 100: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is below a second threshold, directing the renderengine to render the virtual content with the render camera, wherein therender camera has an aperture at the determined position of the centerof perspective of said eye, and wherein the first threshold isindicative of a higher level change with time in the determined positionof the center of perspective of said user's eye than the secondthreshold.

Example 101: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is below a second threshold, directing the renderengine to render the virtual content with the render camera, wherein therender camera has an aperture at the determined position of the centerof perspective of said eye.

Example 102: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is between the first and second thresholds, directingthe render engine to render the virtual content with the render camera,wherein the render camera has an aperture at a point along a linebetween (i) the determined position of the center of rotation of saideye and (ii) the determined position of the center of perspective ofsaid eye.

Example 103: The method of any of the Examples above, furthercomprising:

with at least a portion of said display, said portion being transparentand disposed at a location in front of the user's eye when the userwears said head-mounted display, transmitting light from a portion ofthe environment in front of the user and said head-mounted display tothe user's eye to provide a view of said portion of the environment infront of the user and said head-mounted display.

Example 104: The method of any of the Examples above, furthercomprising:

with the one or more eye tracking cameras, determining a position of atleast one of said iris, pupil, or lens

Example 105: The method of any of the Examples above, furthercomprising:

with the render engine, rendering the virtual image content with therender camera, said render camera configured to present virtual imagesto said eye images that are rendered as if captured by a camera havingan aperture at a position along a line between (i) the determinedposition of the center of rotation of said eye and (ii) the determinedposition of the at least one of said iris or pupil.

Example 106: The method of any of the Examples above, furthercomprising:

with the one or more eye tracking cameras, determining a position of acenter of perspective of said user's eye, wherein the center ofperspective of said user's eye is located less than approximately 1.0 mmfrom the pupil of said user's eye; and

with the render engine, rendering the virtual image content with therender camera, said render camera configured to present virtual imagesto said eye images that are rendered as if captured by a camera havingan aperture at the determined position of the center of perspective ofsaid user's eye.

Example 107: The method of any of the Examples above, furthercomprising:

with the render engine, rendering the virtual image content with therender camera, said render camera configured to present virtual imagesto said eye images that are rendered as if captured by a camera havingan aperture at a position along a line between (i) the determinedposition of the center of rotation of said eye and (ii) the determinedposition of the center of perspective of said user's eye.

Example 108: The method of any of the Examples above, furthercomprising:

with processing electronics in communication with the one or more eyetracking cameras, determining a measure of change with time of thedetermined position of the center of perspective of said user's eye; and

with the processing electronics, if it is determined that the measure ofchange with time exceeds a first threshold, directing the render engineto render the virtual content with the render camera as if captured by acamera having an aperture at the determined position of the center ofrotation of said eye.

Example 109: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is below a second threshold, directing the renderengine to render the virtual content with the render camera as ifcaptured by a camera having an aperture at the determined position ofthe center of perspective of said eye, wherein the first threshold isindicative of a higher level change with time in the determined positionof the center of perspective of said user's eye than the secondthreshold.

Example 110: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is below a second threshold, directing the renderengine to render the virtual content with the render camera as ifcaptured by a camera having an aperture at the determined position ofthe center of perspective of said eye.

Example 111: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is between the first and second thresholds, directingthe render engine to render the virtual content with the render cameraas if captured by a camera having an aperture at a point along a linebetween (i) the determined position of the center of rotation of saideye and (ii) the determined position of the center of perspective ofsaid eye.

Example 112: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, and a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field at different amounts of divergence and thus thedisplayed virtual image content appears to originate from differentdepths at different periods of time, wherein said head-mounted displayis configured to project light into said user's eye having a firstamount of divergence at a first period of time and is configured toproject light into said user's eye having a second amount of divergenceat a second period of time, wherein the first amount of divergence isdifferent from the second amount of divergence;

one or more eye tracking cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore eye tracking cameras, the processing electronics configured toobtain an estimate of a center of rotation of said eye based on imagesof said eye obtained with said one or more eye tracking cameras, obtainan estimate of a vergence distance of said user based on images of saideye obtained with said one or more eye tracking cameras, and shift fromprojecting light into said user's eye at the first amount of divergenceto projecting light into said user's eye at the second amount ofdivergence based on the estimated vergence distance of said user.

Example 113: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 114: The display system of any of the Examples above, whereinsaid processing electronics is further configured to, based on images ofsaid eye obtained with said one or more eye tracking cameras, detect ablink of said eye.

Example 115: The display system of any of the Examples above, whereinsaid processing electronics is further configured to, based on images ofsaid eye obtained with said one or more eye tracking cameras, detect asaccade of said eye.

Example 116: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence based onthe determined vergence distance of said user and based on whether theprocessing electronics have detected the blink of said eye.

Example 117: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence based onthe determined vergence distance of said user and based on whether theprocessing electronics have detected the saccade of said eye.

Example 118: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence based onthe determined vergence distance of said user and based on whether theprocessing electronics have detected at least one of the saccade or theblink of said eye.

Example 119: The display system of any of the Examples above, whereinsaid first amount of divergence is associated with vergence distances ina first range and wherein said second amount of divergence is associatedwith vergence distances in a second range.

Example 120: The display system of any of the Examples above, whereinsaid first amount of divergence is associated with vergence distances ina first range, wherein said second amount of divergence is associatedwith vergence distances in a second range and wherein the first andsecond ranges overlap but are not equal.

Example 121: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the firstrange and lies within the second range.

Example 122: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the secondrange and lies within the first range.

Example 123: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the firstrange and lies within the second range and also detecting a blink ofsaid eye.

Example 124: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the firstrange and lies within the second range and also detecting a saccade ofsaid eye.

Example 125: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user has been outside thefirst range and within the second range for longer than a predeterminedperiod of time.

Example 126: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user has been outside thefirst range and within the second range for longer than a predeterminedperiod of time of at least 10 seconds.

Example 127: The display system of any of the Examples above, whereinsaid head-mounted display comprises a first display element configuredto project light having the first amount of divergence and a seconddisplay element configured to project light having the second amount ofdivergence.

Example 128: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a discrete display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using only one of the first display element.

Example 129: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a blended display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using both of the first and second display elementsfor each of the frames.

Example 130: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a blended display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using both of the first and second display elementsfor each of the frames and wherein, in the blended display mode, thedisplay is configured to project light, using the first and seconddisplay elements, that is perceived by a user as having a given amountof divergence that is between the first and second amounts ofdivergence.

Example 131: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a multi-focus display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using both of the first and second display elementsfor each of the frames, wherein, in the multi-focus display mode, thedisplay is configured to project light associated with first virtualimage content at a third amount of divergence and to project lightassociated with second virtual image content at a fourth amount ofdivergence, and wherein the third amount of divergence is different fromthe fourth amount of divergence.

Example 132: The display system of any of the Examples above, whereinthird and fourth amounts of divergence are each between the first andsecond amounts of divergence.

Example 133: The display system of any of the Examples above, wherein atleast one of the third and fourth amounts of divergence are between thefirst and second amounts of divergence.

Example 134: The display system of any of the Examples above, whereinthe third and fourth amounts of divergence are respectively equal to thefirst and second amounts of divergence.

Example 135: The display system of any of the Examples above, whereinthe display is configured to project light associated with the firstvirtual image in a first region of the user's vision field and toproject light associated with the second virtual image in a secondregion of the user's vision field and wherein the first and secondregions are different.

Example 136: The display system of any of the Examples above, whereinthe display is configured to project light associated with the firstvirtual image in a first region of the user's vision field and toproject light associated with the second virtual image in a secondregion of the user's vision field and wherein the first and secondregions do not overlap.

Example 137: A display system configured to project light to left andright eyes of a user to display virtual image content in a vision fieldof said user, each of said eyes having a cornea, an iris, a pupil, alens, a retina, and an optical axis extending through said lens, pupil,and cornea, said display system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's left and right eyes to display virtualimage content to the user's vision field at different amounts of atleast one of divergence and collimation and thus the displayed virtualimage content appears to originate from different distances from theuser's left and right eyes at different periods of time;

a first eye tracking camera configured to image the user's left eye;

a second eye tracking camera configured to image the user's right eye;and

processing electronics in communication with the display and the firstand second eye tracking cameras, the processing electronics configuredto obtain an estimate of an interpupillary distance between the user'sleft and right eyes based on images of said left and right eyes obtainedwith said first and second eye tracking cameras.

Example 138: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or more eyetracking cameras forming images of said eye using said light from saidone or more light sources.

Example 139: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 140: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 141: The display system of any of the Examples above, whereinone or more light sources form one or more glints on said eye and saidprocessing electronics is configured to determine a location of saidcornea based on said one or more glints.

Example 142: The display system of any of the Examples above, whereinsaid cornea has associated therewith a cornea sphere having a center ofcurvature and said processing electronics is configured to determine alocation of said center of curvature of said cornea sphere.

Example 143: The display system of any of the Examples above, whereinsaid cornea has associated therewith a cornea sphere having a center ofcurvature and said processing electronics is configured to determine alocation of said center of curvature of said cornea sphere based on saidone or more glints.

Example 144: The display system of any of the Examples above, whereinsaid one or more eye tracking camera is configured to image said pupilof said eye.

Example 145: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the location ofsaid center of said pupil.

Example 146: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine at least aportion of a boundary between said iris and said pupil.

Example 147: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a center of saidboundary between said iris and said pupil.

Example 148: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of said pupil in three-dimensional space with respect to acenter of curvature of said cornea.

Example 149: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of said optical axis.

Example 150: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of said optical axis based on a location of said center ofsaid pupil in three-dimensional space.

Example 151: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine said location andorientation of said optical axis based on a location of said center ofsaid pupil in three-dimensional space with respect to a center ofcurvature of said cornea.

Example 152: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea.

Example 153: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea and a location and orientation of said optical axis.

Example 154: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the location ofsaid center of rotation of said eye by translating a particular distancealong said optical axis from said center of curvature of said cornea.

Example 155: A method of rendering virtual image content in a displaysystem configured to project light to left and right eyes of a user todisplay the virtual image content in a vision field of said user, eachof said eyes having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, said methodcomprising:

with one or more eye tracking cameras configured to image said eyes ofthe user to track movements of said eyes, determining a position of acenter of rotation of said left eye and a position of a center ofrotation of said right eye;

with processing electronics in communication with the one or more eyetracking cameras, estimating said user's interpupillary distance basedon the determined positions of the center of rotation of said left andright eyes;

with the one or more eye tracking cameras, determining a current lefteye pose and a current right eye pose; and

with the processing electronics, estimating said user's current vergencedistance by comparing said estimated interpupillary distance and saiddetermined current left eye pose and said determined current right eyepose.

Example 156: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said pupil ofsaid user's left eye and a position of said pupil of said user's righteye.

Example 157: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said corneaof said user's left eye and a position of said cornea of said user'sright eye.

Example 158: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said iris ofsaid user's left eye and a position of said iris of said user's righteye.

Example 159: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said lens ofsaid user's left eye and a position of said lens of said user's righteye.

Example 160: The method of any of the Examples above, wherein estimatingsaid user's current vergence distance comprises:

with processing electronics, estimating a distance between saidpositions of said irises of said user's left and right eyes; and

with the processing electronics, estimating said user's current vergencedistance based on a comparison of said estimated interpupillary distanceand said estimated distance between said positions of said irises ofsaid user's left and right eyes.

Example 161: The method of any of the Examples above, furthercomprising: with a head-mounted display, projecting light into saiduser's eye to display the rendered virtual image content to the user'svision field at different amounts of divergence such that the virtualimage content appears to originate from different depths at differentperiods of time.

Example 162: The method of any of the Examples above, furthercomprising: with at least a portion of said display, said portion beingtransparent and disposed at a location in front of the user's eye whenthe user wears said head-mounted display, transmitting light from aportion of the environment in front of the user and said head-mounteddisplay to the user's eye to provide a view of said portion of theenvironment in front of the user and said head-mounted display.

Example 163: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field at different amounts of at least one ofdivergence and collimation and thus the displayed virtual image contentappears to originate from different depths at different periods of time;

one or more eye tracking cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore eye tracking cameras, the processing electronics configured toobtain a position estimate of a center of rotation of said eye based onimages of said eye obtained with said one or more eye tracking camerasand configured to obtain a direction estimate of the optical axis ofsaid eye based on said images,

wherein said processing electronics is configured to present saidvirtual image content to said user's eye that are rendered as ifcaptured by a camera having an aperture disposed along the optical axisand spaced apart from the estimated position of the center of rotationof said eye by between 6.0 mm and 13.0 mm in a direction away from saidretina.

Example 164: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between7.0 mm and 12.0 mm in a direction away from said retina.

Example 165: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between8.0 mm and 11.0 mm in a direction away from said retina.

Example 166: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between9.0 mm and 10.0 mm in a direction away from said retina.

Example 167: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between9.5 mm and 10.0 mm in a direction away from said retina.

Example 168: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye byapproximately 9.7 mm.

Example 169: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 170: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 171: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 172: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 173: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or more eyetracking cameras capturing images of said eye using said light from saidone or more light sources.

Example 174: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 175: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least three light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 176: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 177: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field at different amounts of at least one ofdivergence and collimation and thus the displayed virtual image contentappears to originate from different depths at different periods of time;

one or more eye tracking cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore eye tracking cameras,

wherein said processing electronics is configured to present saidvirtual image content to said user's eye that are rendered by a rendercamera located at the pupil of the eye or between the pupil and thecornea of the eye.

Example 178: The display system of any of the Examples above, whereinthe render camera is located at a position that is between 1.0 mm and2.0 mm in front of said pupil of said user's eye.

Example 179: The display system of any of the Examples above, whereinthe render camera is located a position that is about 1.0 mm in front ofsaid pupil of said user's eye.

Example 180: The display system of any of the Examples above, whereinthe render camera is located at a position that is between 0.25 mm and1.0 mm in front of said pupil of said user's eye.

Example 181: The display system of any of the Examples above, whereinthe render camera is located at a position that is between 0.5 mm and1.0 mm in front of said pupil of said user's eye.

Example 182: The display system of any of the Examples above, whereinthe render camera is located at position that is between 0.25 mm and 0.5mm in front of said pupil of said user's eye.

Example 183: The display system of any of the Examples above, whereinthe render camera is located at the pupil of the eye.

Example 184: The display system of any of the Examples above, whereinthe render camera is not located at the pupil of the eye.

Example 185: Any of the Examples above, wherein the camera comprises apinhole camera.

Example 186: Any of the Examples above, wherein the aperture comprises apinhole of a pinhole camera.

Example 187: A method of rendering virtual image content in a displaysystem configured to project light to an eye of a user to display thevirtual image content in a vision field of said user, said eye having acornea, an iris, a pupil, a lens, a retina, and an optical axisextending through said lens, pupil, and cornea, said method comprising:

with one or more eye cameras configured to image said eye of the user,determining a position based on said imaging of said eye with said oneor more cameras;

with a render engine, rendering virtual image content with a rendercamera at a location based on said determined position, said rendercamera configured to render virtual images to be presented to said eye;and with a head-mounted display, projecting light into said user's eyeto display the rendered virtual image content to the user's visionfield.

Example 188: The method of any of the Examples above, wherein saidposition is a center of rotation of said eye.

Example 189: The method of any of the Examples above, wherein saidlocation of said render camera is at said center of rotation of saideye.

Example 190: The method of any of the Examples above, wherein saidposition is a center of perspective of said eye.

Example 191: The method of any of the Examples above, wherein saidlocation of said render camera is at said center of perspective of saideye.

Example 192: The method of any of the Examples above, wherein saidrender camera is configured to render virtual images to be presented tosaid eye that are rendered as if captured by a camera having an aperturecloser to said the center of rotation than said retina of said eye.

Example 193: The method of any of the Examples above, wherein saidrender camera is configured to render virtual images to be presented tosaid eye that are rendered as if captured by a camera having an apertureat said the center of rotation.

Example 194: The method of any of the Examples above, wherein saidrender camera is modeled with an aperture at said center of rotation ofsaid eye.

Example 195: The method of any of the Examples above, wherein saidrender camera is modeled with an aperture, a lens, and a detector.

Example 196: The method of any of the Examples above, wherein saidrender camera has an aperture at a position along a line between (i) thedetermined position of the center of rotation of said eye and (ii) thedetermined position of the at least one of said iris or pupil.

Example 197: The method of any of the Examples above, furthercomprising:

with the one or more cameras, determining a position of a center ofperspective of said user's eye, wherein the center of perspective ofsaid user's eye is located less than approximately 1.0 mm from the pupilof said user's eye; and with the render engine, rendering the virtualimage content with the render camera, wherein said render camera has anaperture at the determined position of the center of perspective of saiduser's eye.

Example 198: The method of any of the Examples above, furthercomprising:

with the render engine, rendering the virtual image content with therender camera, wherein said render camera has an aperture at a positionalong a line between (i) the determined position of the center ofrotation of said eye and (ii) the determined position of the center ofperspective of said user's eye.

Example 199: The method of any of the Examples above, furthercomprising:

with processing electronics in communication with the one or morecameras, determining a measure of change with time of the determinedposition of the center of perspective of said user's eye; and

with the processing electronics, if it is determined that the measure ofchange with time exceeds a first threshold, directing the render engineto render the virtual content with the render camera, wherein the rendercamera has an aperture at the determined position of the center ofrotation of said eye.

Example 200: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is below a second threshold, directing the renderengine to render the virtual content with the render camera, wherein therender camera has an aperture at the determined position of the centerof perspective of said eye, and wherein the first threshold isindicative of a higher level change with time in the determined positionof the center of perspective of said user's eye than the secondthreshold.

Example 201: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is below a second threshold, directing the renderengine to render the virtual content with the render camera, wherein therender camera has an aperture at the determined position of the centerof perspective of said eye.

Example 202: The method of any of the Examples above, furthercomprising:

with the processing electronics, if it is determined that the measure ofchange with time is between the first and second thresholds, directingthe render engine to render the virtual content with the render camera,wherein the render camera has an aperture at a point along a linebetween (i) the determined position of the center of rotation of saideye and (ii) the determined position of the center of perspective ofsaid eye.

Example 203: The method of any of the Examples above, furthercomprising:

with at least a portion of said display, said portion being transparentand disposed at a location in front of the user's eye when the userwears said head-mounted display, transmitting light from a portion ofthe environment in front of the user and said head-mounted display tothe user's eye to provide a view of said portion of the environment infront of the user and said head-mounted display.

Example 204: The method of any of the Examples above, furthercomprising:

with the one or more cameras, determining a position of at least one ofsaid iris, pupil, or lens

Example 205: The method of any of the Examples above, furthercomprising:

with the render engine, rendering the virtual image content with therender camera, said render camera configured to present virtual imagesto said eye images that are rendered as if captured by a camera havingan aperture at a position along a line between (i) the determinedposition of the center of rotation of said eye and (ii) the determinedposition of the at least one of said iris or pupil.

Example 206: The method of any of the Examples above, furthercomprising:

with the one or more cameras, determining a position of a center ofperspective of said user's eye, wherein the center of perspective ofsaid user's eye is located less than approximately 1.0 mm from the pupilof said user's eye; and

with the render engine, rendering the virtual image content with therender camera, said render camera configured to present virtual imagesto said eye images that are rendered as if captured by a camera havingan aperture at the determined position of the center of perspective ofsaid user's eye.

Example 207: The method of any of the Examples above, furthercomprising:

with the render engine, rendering the virtual image content with therender camera, said render camera configured to present virtual imagesto said eye images that are rendered as if captured by a camera havingan aperture at a position along a line between (i) the determinedposition of the center of rotation of said eye and (ii) the determinedposition of the center of perspective of said user's eye.

Example 208: The method of any of the Examples above, wherein with saidhead-mounted display, projecting light into said user's eye to displaythe rendered virtual image content to the user's vision field comprisesprojecting light into said user's eye to display the rendered virtualimage content to the user's vision field at different amounts ofdivergence such that the virtual image content appears to originate fromdifferent depths at different periods of time.

Example 209: The method of any of the Examples above, wherein saiddifferent amount of divergence includes zero divergence.

Example 210: The method of any of the Examples above, wherein saiddifferent amount of divergence includes collimation.

Example 211: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field;

one or more cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore cameras, the processing electronics configured to obtain a positionof said eye based on images of said eye obtained with said one or morecameras,

wherein said processing electronics is configured to present saidvirtual image content to said user's eye that are rendered by a rendercamera located at location based on said determined position.

Example 212: The display system of any of the Examples above, whereinsaid position is an estimate of a center of rotation of said eye.

Example 213: The display system of any of the Examples above, whereinsaid location of said render camera is at said estimated center ofrotation of said eye.

Example 214: The display system of any of the Examples above, whereinsaid position is an estimate of a center of perspective of said eye.

Example 215: The display system of any of the Examples above, whereinsaid location of said render camera is at said estimated center ofperspective of said eye.

Example 216: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than saidretina.

Example 217: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of rotation than said retina.

Example 218: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture at said center of rotation.

Example 219: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of rotation than said center ofperspective.

Example 220: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than saidretina.

Example 221: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than a center ofrotation of the eye.

Example 222: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture at said center of perspective.

Example 223: The display system of any of the Examples above, whereinsaid center of perspective is not located at said pupil of said eye.

Example 224: The display system of any of the Examples above, whereinthe processing electronics is configured to obtain an estimate of saiduser's eye pose over time and wherein the processing electronics adjustthe position of the render camera based at least in part upon the user'seye pose.

Example 225: The display system of any of the Examples above, whereinthe processing electronics is configured to track said user's eye poseover time and wherein the position of the render camera is adjusted overtime in response to changes in said user's eye pose over time.

Example 226: The display system of any of the Examples above, whereinthe processing electronics is configured to obtain the estimate of thecenter of perspective by filtering a plurality of estimated center ofperspective positions.

Example 227: The display system of any of the Examples above, whereinthe processing electronics is configured to obtain the estimate of thecenter of perspective by averaging and/or applying a Kalman filter to aplurality of estimated center of perspective positions.

Example 228: The display system of any of the Examples above, whereinthe center of perspective comprises a position within the anteriorchamber of said user's eye.

Example 229: The display system of any of the Examples above, whereinthe center of perspective comprises a position in front of said pupil ofsaid user's eye.

Example 230: The display system of any of the Examples above, whereinthe center of perspective comprises a position that is between 1.0 mmand 2.0 mm in front of said pupil of said user's eye.

Example 231: The display system of any of the Examples above, whereinthe center of perspective comprises a position that is about 1.0 mm infront of said pupil of said user's eye.

Example 232: The display system of any of the Examples above, whereinthe center of perspective comprises a position that is between 0.25 mmand 1.0 mm in front of said pupil of said user's eye.

Example 233: The display system of any of the Examples above, whereinthe center of perspective comprises a position that is between 0.5 mmand 1.0 mm in front of said pupil of said user's eye.

Example 234: The display system of any of the Examples above, whereinthe center of perspective comprises a position that is between 0.25 mmand 0.5 mm in front of said pupil of said user's eye.

Example 235: The display system of any of the Examples above, whereinthe center of perspective lies along the optical axis of said eye andwherein said processing electronics are further configured to obtain theposition estimate of the center of perspective by obtaining a positionestimate of the optical axis of said eye.

Example 236: The display system of any of the Examples above, whereinthe center of perspective lies along the optical axis of said eye at aposition between an outer surface of the cornea and the pupil of saideye and wherein said processing electronics are further configured toobtain the position estimate of the center of perspective by obtaining aposition estimate of the optical axis of said eye.

Example 237: The display system of any of the Examples above, whereinthe center of perspective lies along the optical axis of said eye at aposition between an outer surface of the cornea and the pupil of saideye and wherein said processing electronics are further configured toobtain the position estimate of the center of perspective by obtaining aposition estimate of the optical axis of said eye and a positionestimate of a center of rotation of said eye, the cornea of said eye,the iris of said eye, the retina of said eye, and the pupil of said eyeor any combinations thereof.

Example 238: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 239: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 240: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 241: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 242: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or morecameras capturing images of said eye using said light from said one ormore light sources.

Example 243: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 244: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least three light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 245: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 246: The display system of any of the Examples above, whereinsaid one or more light sources form one or more glints on said eye andsaid processing electronics is configured to determine the position ofthe center of curvature of said cornea based on said one or more glints.

Example 247: The display system of any of the Examples above, whereinsaid one or more light sources form one or more glints on said eye andsaid processing electronics is configured to determine athree-dimensional position of the center of curvature of said corneabased on said one or more glints.

Example 248: The display system of any of the Examples above, whereinsaid one or more cameras are further configured to image said pupil ofthe user's eye and wherein said processing electronics are furtherconfigured to determine the position of said pupil of said eye based atleast on the image of said pupil from the one or more cameras.

Example 249: The display system of any of the Examples above, whereinsaid one or more cameras are further configured to image said pupil ofthe user's eye and wherein said processing electronics are furtherconfigured to determine a three-dimensional position of said pupil ofsaid eye based at least on the image of said pupil from the one or morecameras.

Example 250: The display system of any of the Examples above, whereinsaid one or more cameras are further configured to image said pupil ofthe user's eye and wherein said processing electronics are furtherconfigured to determine the position of said pupil of said eye based onthe position of the center of curvature of said cornea and based on theimage of said pupil from the one or more cameras.

Example 251: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the optical axisof said eye based on the three-dimensional position of the center ofcurvature of said cornea and based on the three-dimensional position ofsaid pupil.

Example 252: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a visual axis ofsaid eye based on the optical axis.

Example 253: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the visual axisof said eye based on the optical axis and the three-dimensional positionof at least one of the center of curvature of said cornea, said pupil orboth.

Example 254: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine athree-dimensional position of the center of rotation of said eye basedon the three-dimensional position of the center of curvature of saidcornea.

Example 255: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine athree-dimensional position of the center of rotation of said eye basedon the three-dimensional position of the center of curvature of saidcornea and based on said optical axis.

Example 256: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the distancebetween said eye and the additional eye of said user based at least onthe three-dimensional position of the center of rotation of said eye.

Example 257: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine an interpupillarydistance between said eye and the additional eye of said user based atleast on the three-dimensional position of the center of rotation ofsaid eye.

Example 258: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the optical axis of said eye.

Example 259: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the optical axis of said eye andon a determined optical axis of the additional eye of said user.

Example 260: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the visual axis of said eye andon a determined visual axis of the additional eye of said user.

Example 261: The display system of any of the Examples above, whereinsaid display is configured to project collimated light into said user'seye.

Example 262: The display system of any of the Examples above, whereinsaid display is configured to project collimated light corresponding toan image pixel into said user's eye at a first period of time anddivergent light corresponding to said image pixel into said user's eyeat a second period of time.

Example 263: The display system of any of the Examples above, whereinsaid display is configured to project light corresponding to an imagepixel having a first amount of divergence into said user's eye at afirst period of time and to project light corresponding to said imagepixel having a second amount of divergence, greater than the firstamount of divergence, into said user's eye at a second period of time.

Example 264: The display system of any of the Examples above, whereincenter of perspective is estimated to be proximal said pupil of said eye

Example 265: The display system of any of the Examples above, whereincenter of perspective is estimated to be between said cornea and saidpupil of said eye.

Example 266: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths at different periods of time.

Example 267: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field at different amounts of at least one ofdivergence and collimation and thus the displayed virtual image contentappears to originate from different depths at different periods of time;

one or more eye tracking cameras configured to image the user's eye; and

processing electronics in communication with the display and the one ormore eye tracking cameras, the processing electronics configured to:

-   -   obtain a position and orientation estimate of an optical axis        extending through the lens, pupil, and cornea of the user's eye        based on images of said eye obtained with one or more eye        tracking cameras;    -   identify a particular location along an axis in render space        that is registered to said optical axis of said eye; and

present said virtual image content to said user's eye as rendered by avirtual render camera located at the particular location in renderspace.

Example 268: The display system of Examples 267, wherein saidhead-mounted display is configured to project light into said user's eyehaving a first amount of divergence at a first period of time and isconfigured to project light into said user's eye having a second amountof divergence at a second period of time, wherein the first amount ofdivergence is different from the second amount of divergence, andwherein the processing electronics are further configured to obtain anestimate of a vergence distance of said user based on images of said eyeobtained with said one or more eye tracking cameras, and shift fromprojecting light into said user's eye at the first amount of divergenceto projecting light into said user's eye at the second amount ofdivergence based on the estimated vergence distance of said user.

Example 269: The display system of Examples 267, wherein saidhead-mounted display is configured to project light into said user's eyehaving a first amount of divergence at a first period of time and isconfigured to project light into said user's eye having a second amountof divergence at a second period of time, wherein the first amount ofdivergence is different from the second amount of divergence, andwherein the processing electronics are further configured to obtain anestimate of a vergence distance of said user based on images of said eyeobtained with said one or more eye tracking cameras, and shift fromprojecting light into said user's eye at the first amount of divergenceto projecting light into said user's eye at the second amount ofdivergence based on the estimated vergence distance of said user.

Example 270: The display system of Example 267, wherein the processingelectronics are further configured to determine a position along theoptical axis at which a center of rotation of said eye is estimated tobe located based on images of said eye obtained with one or more eyetracking cameras, and

wherein the processing electronics are configured to identify theparticular location along the axis in render space that is registered tosaid optical axis of said eye based on said position along the opticalaxis at which the center of rotation of said eye is estimated to belocated.

Example 271: The display system of Examples 267, wherein the particularlocation along the axis in render space comprises a location along theaxis in render space at which parallax shifts in the render space aredetermined to be reduced.

Example 272: The display system of Examples 267, wherein the particularlocation along the axis in render space comprises a location along theaxis in render space at which parallax shifts in the render space aredetermined to be minimized.

Example 273: The display system of any of the Examples above, whereinsaid processing electronics are configured to obtain the estimate of thecenter of rotation of said eye based on determination of multiple gazedirections of the user's eye over a period of time during which said eyeis rotating based on images of said eye obtained by said one or more eyetracking cameras.

Example 274: The display system of Example 273, wherein said processingelectronics are configured to determine said gaze direction based onvariations of the shape of one or more of the pupil, iris, or limbus ofthe user's eye in images obtained with said one or more eye trackingcameras over a period of time during which said eye is rotating.

Example 275: The display system of any of the Examples above, whereinsaid processing electronics are configured to determine an array ofpositions based on a plurality of spatial locations on an image of theuser's eye obtained with said one or more eye tracking cameras.

Example 276: The display system of Example 275, wherein said array ofpositions corresponds to at least a portion of an ellipse.

Example 277: The display system of any of Examples 275 or 276, whereinsaid processing electronics are configured to determine said array ofpositions by fitting a curve to said plurality of spatial locations onsaid image of the user's eye.

Example 278: The display system of Example 277, wherein said curvecomprises an ellipse.

Example 279: The display system of any of Examples 275 to 278, whereinsaid plurality of spatial locations on said image comprises spatiallocations on the limbus of said user's eye in said image.

Example 280: The display system of any of Examples 275 to 279, whereinsaid plurality of spatial locations on said image comprises spatiallocations on a boundary between the iris and the sclera of said user'seye in said image.

Example 281: The display system of any of Examples 275 to 279, whereinsaid plurality of spatial locations on said image comprises spatiallocations on a boundary between the cornea and the sclera of said user'seye in said image obtained with said one or more eye tracking cameras.

Example 282: The display system of any of Examples 275 to 281, whereinsaid processing electronics are configured to determine a plurality oflinear paths extending from a location on a first side of said array ofpositions through said array of positions to a second opposite side ofsaid array of positions.

Example 283: The display system of Example 282, wherein said processingelectronics are configured to determine a circular region based on saidplurality of linear paths, said circular region having a radius, R.

Example 284: The display system of Example 283, wherein said radius, R,corresponds to an average radius of a limbus.

Example 285: The display system of Example 283, wherein said radius, R,corresponds to a measured radius of the limbus of the user's eye.

Example 286: The display system of Example 283, wherein said radius, R,corresponds to an average radius of a pupil.

Example 287: The display system of Example 283, wherein said radius, R,corresponds to a measured radius of the pupil of the user's eye.

Example 288: The display system of any of Examples 282 to 287, whereinsaid processing electronics are configured to determine the location anddirection of a normal through a central portion of said circular region.

Example 289: The display system of any of Examples 282 to 288, whereinsaid processing electronics are configured to determine respectivelocations and directions of a plurality of normals through centralportions of respective circular regions based on a plurality of imagesof said eye previously obtained with said one or more eye trackingcameras.

Example 290: The display system of Example 289, wherein said processingelectronics are configured to determine a position where said pluralityof normals converge or intersect.

Example 291: The display system of Example 289, wherein said processingelectronics are configured to obtain the estimate of the center ofrotation of said user's eye by identifying a region of intersection,convergence, or close proximity of multiple of said normals determinedbased on images of the user's eye obtained over a period of time duringwhich said eye is rotating.

Example 292: The display system of Example 289, wherein said processingelectronics are configured to obtain the estimate of the center ofrotation of said user's eye based on the locations and directions ofmultiple of said plurality of normals determined based on images of theuser's eye obtained over a period of time during which said eye isrotating.

Example 293: The display system of any of Examples 282-292, wherein thelocation on the first side of said array positions corresponds to anorigin of a coordinate system of one of said one or more eye trackingcameras.

Example 294: The display system of any of the Examples above, whereinthe processing electronics is configured to obtain the estimate of thecenter of rotation by filtering, averaging, applying a Kalman filter, orany combinations thereof to a plurality of estimated center of rotationpositions.

Various additional examples of display systems that project light to oneor more eyes of a user to display virtual image content in a visionfield of said user are described herein such as the additional examplesenumerated below:

Additional Example 1: A display system configured to project light to aneye of a user to display virtual image content in a vision field of saiduser, said eye having a cornea, an iris, a pupil, a lens, a retina, andan optical axis extending through said lens, pupil, and cornea, saiddisplay system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field, at least a portion of said display beingtransparent and disposed at a location in front of the user's eye whenthe user wears the frame such that said transparent portion transmitslight from a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display;

an inward-facing imaging system configured to image the user's eye; and

processing electronics in communication with the inward-facing imagingsystem, the processing electronics configured to obtain an estimate of acenter of rotation of said eye based on multiple images of said eyeobtained with said inward-facing imaging system, said processingelectronics configured to a determine a variation in calculated valuesof center of rotation and select a statistically determined estimate ofcenter of rotation based on said variation.

Additional Example 2: The display system of Additional Example 1,wherein reduced variation is used to identify said statisticallydetermined center of rotation.

Additional Example 3: The display system of Additional Example 1 or 2,wherein a first set of estimates of center of rotation are calculatedbased on a first value of a parameter used to calculate the center ofrotation and a first variation is determined from said first set ofestimates.

Additional Example 4: The display system of Additional Example 3,wherein a second set of estimates of center of rotation are calculatedbased on a second value of said parameter and a second variation isdetermined from said second set of estimates.

Additional Example 5: The display system of Additional Example 4,wherein said first and second variations are compared to determine whichset has reduced variation, said determination of said statisticallydetermined estimate of the center of rotation being based on thiscomparison.

Additional Example 6: The display system of Additional Example 1 or 2,wherein multiple sets of values of centers of rotation are calculatedbased on multiple respective values of a parameter and respectivevariations determined for the different respective sets.

Additional Example 7: The display system of Additional Example 6,wherein the respective variations are compared to determine which sethas reduced variation and said determination of said statisticallydetermined estimate of said center of rotation is based on saidcomparison.

Additional Example 8: The display system of Additional Example 6 or 7,wherein the value of the parameter for said set having the lowestvariation is used in calculating said statistically determined estimateof said center of rotation.

Additional Example 9: The display system of Additional Example 6, 7, or8, wherein said set having the lowest the variation is used incalculating said statistically determined estimate of said center ofrotation.

Additional Example 10: The display system of any of Additional Examples3 to 9, wherein said parameter comprises distance to said center ofrotation from the center of curvature of said cornea.

Additional Example 11: The display system of any of Additional Examples3 to 9, wherein said parameter comprises distance along the optical axisfrom the center of curvature of said cornea to said center of rotation.

Additional Example 12: The display system of any of the AdditionalExamples above, wherein said variation comprises variance and/orstandard deviation.

Additional Example 13: The display system of any of the AdditionalExamples above, further comprising one or more light sources disposed onsaid frame with respect to said user's eye to illuminate said user'seye, said inward-facing imaging system forming images of said eye usingsaid light from said one or more light sources.

Additional Example 14: The display system of Additional Example 13,wherein said one or more light sources comprises at least two lightsources disposed on said frame with respect to said user's eye toilluminate said user's eye.

Additional Example 15: The display system of Additional Example 13 or14, wherein said one or more light sources comprises infrared lightemitters.

Additional Example 16: The display system of any of the AdditionalExamples 13 to 15, wherein one or more light sources form one or moreglints on said eye and said processing electronics is configured todetermine a location of said cornea based on said one or more glints.

Additional Example 17: The display system of any of the AdditionalExamples 13 to 16, wherein said cornea has associated therewith a corneasphere having a center of curvature and said processing electronics isconfigured to determine a location of said center of curvature of saidcornea sphere.

Additional Example 18: The display system of Additional Example 17,wherein said cornea has associated therewith a cornea sphere having acenter of curvature and said processing electronics is configured todetermine a location of said center of curvature of said cornea spherebased on said one or more glints.

Additional Example 19: The display system of any of the AdditionalExamples above, wherein said inward-facing imaging system is configuredto image said pupil of said eye.

Additional Example 20: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine the location of said center of said pupil.

Additional Example 21: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine at least a portion of a boundary between said iris and saidpupil.

Additional Example 22: The display system of Additional Example 21,wherein said processing electronics is configured to determine a centerof said boundary between said iris and said pupil.

Additional Example 23: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine a location of said center of said pupil in three-dimensionalspace with respect to a center of curvature of said cornea.

Additional Example 24: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine a location and orientation of said optical axis.

Additional Example 25: The display system of Additional Example 24,wherein said processing electronics is configured to determine alocation and orientation of said optical axis based on a location ofsaid center of said pupil in three-dimensional space.

Additional Example 26: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine said location and orientation of said optical axis based on alocation of said center of said pupil in three-dimensional space withrespect to a center of curvature of said cornea.

Additional Example 27: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine a location of said center of rotation of said eye based on acenter of curvature of said cornea.

Additional Example 28: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured todetermine a location of said center of rotation of said eye based on acenter of curvature of said cornea and a location and orientation ofsaid optical axis.

Additional Example 29: The display system of Additional Example 28,wherein said processing electronics is configured to determine thelocation of said center of rotation of said eye by translating aparticular distance along said optical axis from said center ofcurvature of said cornea.

Additional Example 30: The display system of Additional Example 29,wherein said particular distance from said center of curvature to saidcenter of rotation is between 4.0 mm and 6.0 mm

Additional Example 31: The display system of Additional Example 29 or30, wherein said particular distance from said center of curvature tosaid center of rotation is about 4.7 mm.

Additional Example 32: The display system of Additional Example 20 or30, wherein said processing electronics is configured to determine theparticular distance based at least on one or more images of said eyepreviously obtained with said inward-facing imaging system.

Additional Example 33: The display system of any of the AdditionalExamples above, wherein said processing electronics includes electronicson said frame.

Additional Example 34: The display system of any of the AdditionalExamples above, wherein said processing electronics includes electronicson said frame and electronics disposed at a location remote from saidframe.

Additional Example 35: The display system of any of the AdditionalExamples above, wherein said processing electronics includes electronicson said frame and electronics on a belt pack.

Additional Example 36: The display system of any of the AdditionalExamples above, wherein at least a portion of said display istransparent and disposed at a location in front of the user's eye whenthe user wears said head-mounted display such that said transparentportion transmits light from a portion of the environment in front ofthe user and said head-mounted display to the user's eye to provide aview of said portion of the environment in front of the user and saidhead-mounted display.

Additional Example 37: The display system of any of the AdditionalExamples above, wherein said head-mounted display receives light from aportion of the environment in front of the user at a first amount ofdivergence and transmits the light from the portion of the environmentin front of the user to the user's eye with a second amount ofdivergence that is substantially similar to the first amount ofdivergence.

Additional Example 38: The display system of any of the AdditionalExamples above, wherein the processing electronics is configured toobtain the estimate of the center of rotation by filtering, averaging,applying a Kalman filter, or any combinations thereof a plurality ofestimated center of rotation positions.

Additional Example 39: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured topresent said virtual image content to said user's eye that is renderedas if captured by a camera having an aperture at the determined positionof the center of rotation of said user's eye.

Additional Example 40: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured to usea render camera at said center of rotation to render virtual images tobe presented to said eye.

Additional Example 41: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured to usea render camera configured to render virtual images to be presented tosaid eye that are rendered as if captured by a camera having an aperturecloser to said the center of rotation than said retina of said eye.

Additional Example 42: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured to usea render camera configured to render virtual images to be presented tosaid eye that are rendered as if captured by a camera having an apertureat said the center of rotation of said eye.

Additional Example 43: The display system of any of the AdditionalExamples above, wherein said processing electronics is configured to usea render camera at said center of rotation to render virtual images tobe presented to said eye, said render camera modeled with an aperture atsaid center of rotation of said eye.

Additional Example 44: The display system of any of Additional Examples1 to 9, wherein said processing electronics is configured to select thestatistically-determined estimate of center of rotation based on saidvariation during a calibration procedure.

Additional Example 45: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to obtain the estimate of the center of rotation of said eyebased on determination of multiple gaze directions of the user's eyeover a period of time during which said eye is rotating based on imagesof said eye obtained by said one or more eye tracking cameras.

Additional Example 46: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine said gaze direction based on variations of theshape of one or more of the pupil, iris, or limbus of the user's eye inimages obtained with said one or more eye tracking cameras over a periodof time during which said eye is rotating.

Additional Example 47: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine an array of positions based on a plurality ofspatial locations on an image of the user's eye obtained with said oneor more eye tracking cameras.

Additional Example 48: The display system of any of the Examples orAdditional Examples above, wherein said array of positions correspondsto at least a portion of an ellipse.

Additional Example 49: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine said array of positions by fitting a curve tosaid plurality of spatial locations on said image of the user's eye.

Additional Example 50: The display system of any of the Examples orAdditional Examples above, wherein said curve comprises an ellipse.

Additional Example 51: The display system of any of the Examples orAdditional Examples above, wherein said plurality of spatial locationson said image comprises spatial locations on the limbus of said user'seye in said image.

Additional Example 52: The display system of any of the Examples orAdditional Examples above, wherein said plurality of spatial locationson said image comprises spatial locations on a boundary between the irisand the sclera of said user's eye in said image.

Additional Example 53: The display system of any of the Examples orAdditional Examples above, wherein said plurality of spatial locationson said image comprises spatial locations on a boundary between thecornea and the sclera of said user's eye in said image obtained withsaid one or more eye tracking cameras.

Additional Example 54: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine a plurality of linear paths extending from alocation on a first side of said array of positions through said arrayof positions to a second opposite side of said array of positions.

Additional Example 55: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine a circular region based on said plurality oflinear paths, said circular region having a radius, R.

Additional Example 56: The display system of any of the Examples orAdditional Examples above, wherein said radius, R, corresponds to anaverage radius of a limbus.

Additional Example 57: The display system of any of the Examples orAdditional Examples above, wherein said radius, R, corresponds to ameasured radius of the limbus of the user's eye.

Additional Example 58: The display system of any of the Examples orAdditional Examples above, wherein said radius, R, corresponds to anaverage radius of a pupil.

Additional Example 59: The display system of any of the Examples orAdditional Examples above, wherein said radius, R, corresponds to ameasured radius of the pupil of the user's eye.

Additional Example 60: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine the location and direction of a normal through acentral portion of said circular region.

Additional Example 61: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine respective locations and directions of aplurality of normals through central portions of respective circularregions based on a plurality of images of said eye previously obtainedwith said one or more eye tracking cameras.

Additional Example 62: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to determine a position where said plurality of normalsconverge or intersect.

Additional Example 63: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to obtain the estimate of the center of rotation of saiduser's eye by identifying a region of intersection, convergence, orclose proximity of multiple of said normals determined based on imagesof the user's eye obtained over a period of time during which said eyeis rotating.

Additional Example 64: The display system of any of the Examples orAdditional Examples above, wherein said processing electronics areconfigured to obtain the estimate of the center of rotation of saiduser's eye based on the locations and directions of multiple of saidplurality of normals determined based on images of the user's eyeobtained over a period of time during which said eye is rotating.

Additional Example 65: The display system of any of the Examples orAdditional Examples above, wherein the location on the first side ofsaid array positions corresponds to an origin of a coordinate system ofone of said one or more eye tracking cameras.

Additional Example 66: The display system of any of the Examples orAdditional Examples above, wherein the processing electronics isconfigured to obtain the estimate of the center of rotation byfiltering, averaging, applying a Kalman filter, or any combinationsthereof to a plurality of estimated center of rotation positions.

Any of the above Examples or Additional Examples can be combined.Additionally, any of the above Examples or Additional Examples can beintegrated with a head mounted display. In addition, any of the aboveExamples or Additional Examples can be implemented with a single depthplane and/or with one or more variable depth planes (e.g., one or moreelements with variable focusing power that provide accommodation cuesthat vary over time).

Furthermore, apparatus and methods for determining a variety of values,parameters, etc., such as, but not limited to, anatomical, optical, andgeometric features, locations, and orientations, etc., are disclosedherein. Examples of such parameters include, for example, the center ofrotation of the eye, the center of curvature of the cornea, the centerof the pupil, the boundary of the pupil, the center of the iris, theboundary of the iris, the boundary of the limbus, the optical axis ofthe eye, the visual axis of the eye, the center of perspective, but arenot limited to these. Determinations of such values, parameters, etc.,as recited herein include estimations thereof and need not necessarilycoincide precisely with the actual values. For example, determinationsof the center of rotation of the eye, the center of curvature of thecornea, the center or boundary of the pupil or iris, the boundary of thelimbus, the optical axis of the eye, the visual axis of the eye, thecenter of perspective, etc., may be estimations, approximations, orvalues close to, but not the same as, the actual (e.g., anatomical,optical, or geometric) values or parameters. In some cases, for example,root mean square estimation techniques are used to obtain estimates ofsuch values. As an example, certain techniques described herein relateto identifying a location or point at which rays or vectors intersect.Such rays or vectors, however, may not intersect. In this example, thelocation or point may be estimated. For example, the location or pointmay be determined based on root mean square, or other, estimationtechniques (e.g., the location or point may be estimated to be close toor the closest to the rays or vectors). Other processes may also be usedto estimate, approximate or otherwise provide a value that may notcoincide with the actual value. Accordingly, the term determining andestimating, or determined and estimated, are used interchangeablyherein. Reference to such determined values may therefore includeestimates, approximations, or values close to the actual value.Accordingly, reference to determining a parameter or value above, orelsewhere herein should not be limited precisely to the actual value butmay include estimations, approximations or values close thereto.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates example components of a wearablesystem.

FIG. 4 schematically illustrates an example of a waveguide stack of awearable device for outputting image information to a user.

FIG. 5 schematically illustrates an example of an eye.

FIG. 5A schematically illustrates an example coordinate system fordetermining an eye pose of an eye

FIG. 6 is a schematic diagram of a wearable system that includes an eyetracking system.

FIG. 7A is a block diagram of a wearable system that may include an eyetracking system.

FIG. 7B is a block diagram of a render controller in a wearable system.

FIG. 8A is a schematic diagram of an eye showing the eye's cornealsphere.

FIG. 8B illustrates an example corneal glint detected by an eye trackingcamera.

FIGS. 8C-8E illustrate example stages of locating a user's cornealcenter with an eye tracking module in a wearable system.

FIGS. 9A-9C illustrate an example normalization of the coordinate systemof eye tracking images.

FIGS. 9D-9G illustrate example stages of locating a user's pupil centerwith an eye tracking module in a wearable system.

FIG. 9H illustrates example calculated locations of a pupil center thatresults from both taking into account cornea refraction and not takinginto account cornea refraction.

FIGS. 9I-9M illustrate example experimental variations of the calculatedCenter of Rotation of the eye based on different calculated pupilcenters.

FIG. 10 illustrates an example of an eye including the eye's optical andvisual axes and the eye's center of rotation.

FIG. 11 is a process flow diagram of an example of a method for usingeye tracking in rendering content and providing feedback on registrationin a wearable device.

FIG. 12 is a set of example graphs illustrating how the wearable systemmay switch depth planes in response to the user's eye movements.

FIG. 13 depicts a mixed reality system in which certain virtual objectsmay be discretized in depth into one or more depth planes.

FIGS. 14A and 14B illustrate an example in which a mixed reality systemdisplays virtual objects using a single depth plane.

FIGS. 14C and 14D illustrate an example in which a mixed reality systemdisplays virtual objects using two adjacent depth planes to generateaccommodation cues between the two adjacent depth planes.

FIGS. 14E and 14F illustrate an example in which a mixed reality systemdisplays virtual objects using two or more depth planes to generate twoor more accommodation cues simultaneously.

FIG. 15A illustrates an example in which a mixed reality system displaysvirtual objects in the manner of FIGS. 14A and 14B in the presence of acenter of perspective misalignment.

FIG. 15B illustrates an example in which a mixed reality system displaysvirtual objects in the manner of FIGS. 14C and 14D in the presence of acenter of perspective misalignment.

FIG. 15C illustrates an example in which a mixed reality system displaysvirtual objects in the manner of FIGS. 14E and 14F in the presence of acenter of perspective misalignment.

FIGS. 16A and 16B illustrate an example in which the pinhole of a rendercamera is aligned with an eye's center of perspective or approximatelywith an eye's pupil.

FIG. 16C illustrates another example in which the pinhole of a rendercamera is aligned with an eye's center of perspective or approximatelywith an eye's pupil.

FIGS. 17A and 17B illustrate an example in which the pinhole of a rendercamera is aligned with an eye's center of rotation.

FIG. 17C illustrates another example in which the pinhole of a rendercamera is aligned with an eye's center of rotation.

FIGS. 18A and 18B are a set of example graphs illustrating eye trackingdata related to a user's center of perspective.

FIG. 18C is an example graph illustrating how render camera position mayvary as a function of eye tracking data.

FIG. 18D is a schematic diagram of a user's eye illustrating variousrender camera positions that may be utilized as a function of eyetracking data.

FIG. 19 illustrates a graphical representation of the use of an ellipseprojected onto an image of a user's eye (e.g., limbus) used to determinean estimate of the CoR according to various techniques described herein.Rays can be traced through the projection ellipse to form a cone, inwhich circles are fit. In certain implementations, normals through oneor the circles can be used to estimate a CoR.

FIG. 20 is a flowchart of an example process for determining an eye'scenter of rotation based on the eye's limbus.

FIG. 21A illustrates a first ellipse projected onto an image of a user'slimbus (hereinafter referred to as a ‘projection ellipse’) determinedbased on a first gaze of a user. Rays through the projection ellipse areshown that form a cone in a circle is fit.

FIG. 21B illustrates a second projection ellipse determined based on asecond gaze of the user's eye. A second cone and circle fit thereto arealso shown.

FIG. 21C illustrates a third projection ellipse determined based on athird gaze of the user's eye. A third cone and circle fit thereto arealso shown.

FIG. 21D illustrates determining a center of rotation based on thecircles obtained using the projection ellipses as described above.

FIG. 22 illustrates an optical system including two point light sources,an aperture, a lens, and a projection screen.

FIG. 23A illustrates an embodiment of an optical system in a firststage.

FIG. 23B illustrates the optical system in a second stage.

FIG. 24A illustrates another embodiment of an optical system in a firststage.

FIG. 24B illustrates the optical system in a second stage.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout. Unless indicated otherwise, thedrawings are schematic not necessarily drawn to scale.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. At least a portion of the wearable system canbe implemented on a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction. The wearabledevice can be used interchangeably as an AR device (ARD). Further, forthe purpose of the present disclosure, the term “AR” is usedinterchangeably with the term “MR”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1 , an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2 illustrates an example of wearable system 200 which can beconfigured to provide an AR/VR/MR scene. The wearable system 200 canalso be referred to as the AR system 200. The wearable system 200includes a display 220, and various mechanical and electronic modulesand systems to support the functioning of display 220. The display 220may be coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 can be positioned in front of the eyes ofthe user 210. The display 220 can present AR/VR/MR content to a user.The display 220 can comprise a head mounted display (HMD) that is wornon the head of the user.

In some embodiments, a speaker 240 is coupled to the frame 230 andpositioned adjacent the ear canal of the user (in some embodiments,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). The display220 can include an audio sensor (e.g., a microphone) 232 for detectingan audio stream from the environment and capture ambient sound. In someembodiments, one or more other audio sensors, not shown, are positionedto provide stereo sound reception. Stereo sound reception can be used todetermine the location of a sound source. The wearable system 200 canperform voice or speech recognition on the audio stream.

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4 ) which observes the world in the environment aroundthe user. The wearable system 200 can also include an inward-facingimaging system 462 (shown in FIG. 4 ) which can track the eye movementsof the user. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210. The inward-facing imaging system 462 mayinclude one or more cameras. For example, at least one camera may beused to image each eye. The images acquired by the cameras may be usedto determine pupil size or eye pose for each eye separately, therebyallowing presentation of image information to each eye to be dynamicallytailored to that eye.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; orb)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules are availableas resources to the local processing and data module 260. In addition,remote processing module 280 and remote data repository 280 may beoperatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

Example Components of a Wearable System

FIG. 3 schematically illustrates example components of a wearablesystem. FIG. 3 shows a wearable system 200 which can include a display220 and a frame 230. A blown-up view 202 schematically illustratesvarious components of the wearable system 200. In certain implements,one or more of the components illustrated in FIG. 3 can be part of thedisplay 220. The various components alone or in combination can collecta variety of data (such as e.g., audio or visual data) associated withthe user of the wearable system 200 or the user's environment. It shouldbe appreciated that other embodiments may have additional or fewercomponents depending on the application for which the wearable system isused. Nevertheless, FIG. 3 provides a basic idea of some of the variouscomponents and types of data that may be collected, analyzed, and storedthrough the wearable system.

FIG. 3 shows an example wearable system 200 which can include thedisplay 220. The display 220 can comprise a display lens 226 that may bemounted to a user's head or a housing or frame 230, which corresponds tothe frame 230. The display lens 226 may comprise one or more transparentmirrors positioned by the housing 230 in front of the user's eyes 302,304 and may be configured to bounce projected light 338 into the eyes302, 304 and facilitate beam shaping, while also allowing fortransmission of at least some light from the local environment. Thewavefront of the projected light beam 338 may be bent or focused tocoincide with a desired focal distance of the projected light. Asillustrated, two wide-field-of-view machine vision cameras 316 (alsoreferred to as world cameras) can be coupled to the housing 230 to imagethe environment around the user. These cameras 316 can be dual capturevisible light/non-visible (e.g., infrared) light cameras. The cameras316 may be part of the outward-facing imaging system 464 shown in FIG. 4. Image acquired by the world cameras 316 can be processed by the poseprocessor 336. For example, the pose processor 336 can implement one ormore object recognizers 708 (e.g., shown in FIG. 7 ) to identify a poseof a user or another person in the user's environment or to identify aphysical object in the user's environment.

With continued reference to FIG. 3 , a pair of scanned-lasershaped-wavefront (e.g., for depth) light projector modules with displaymirrors and optics configured to project light 338 into the eyes 302,304 are shown. The depicted view also shows two miniature infraredcameras 324 paired with infrared light sources 326 (such as lightemitting diodes “LED”s), which are configured to be able to track theeyes 302, 304 of the user to support rendering and user input. Thecameras 324 may be part of the inward-facing imaging system 462 shown inFIG. 4 . The wearable system 200 can further feature a sensor assembly339, which may comprise X, Y, and Z axis accelerometer capability aswell as a magnetic compass and X, Y, and Z axis gyro capability,preferably providing data at a relatively high frequency, such as 200Hz. The sensor assembly 339 may be part of the IMU described withreference to FIG. 2A The depicted system 200 can also comprise a headpose processor 336, such as an ASIC (application specific integratedcircuit), FPGA (field programmable gate array), or ARM processor(advanced reduced-instruction-set machine), which may be configured tocalculate real or near-real time user head pose from wide field of viewimage information output from the capture devices 316. The head poseprocessor 336 can be a hardware processor and can be implemented as partof the local processing and data module 260 shown in FIG. 2A.

The wearable system can also include one or more depth sensors 234. Thedepth sensor 234 can be configured to measure the distance between anobject in an environment to a wearable device. The depth sensor 234 mayinclude a laser scanner (e.g., a lidar), an ultrasonic depth sensor, ora depth sensing camera. In certain implementations, where the cameras316 have depth sensing ability, the cameras 316 may also be consideredas depth sensors 234.

Also shown is a processor 332 configured to execute digital or analogprocessing to derive pose from the gyro, compass, or accelerometer datafrom the sensor assembly 339. The processor 332 may be part of the localprocessing and data module 260 shown in FIG. 2 . The wearable system 200as shown in FIG. 3 can also include a position system such as, e.g., aGPS 337 (global positioning system) to assist with pose and positioninganalyses. In addition, the GPS may further provide remotely-based (e.g.,cloud-based) information about the user's environment. This informationmay be used for recognizing objects or information in user'senvironment.

The wearable system may combine data acquired by the GPS 337 and aremote computing system (such as, e.g., the remote processing module270, another user's ARD, etc.) which can provide more information aboutthe user's environment. As one example, the wearable system candetermine the user's location based on GPS data and retrieve a world map(e.g., by communicating with a remote processing module 270) includingvirtual objects associated with the user's location. As another example,the wearable system 200 can monitor the environment using the worldcameras 316 (which may be part of the outward-facing imaging system 464shown in FIG. 4 ). Based on the images acquired by the world cameras316, the wearable system 200 can detect objects in the environment(e.g., by using one or more object recognizers 708 shown in FIG. 7 ).The wearable system can further use data acquired by the GPS 337 tointerpret the characters.

The wearable system 200 may also comprise a rendering engine 334 whichcan be configured to provide rendering information that is local to theuser to facilitate operation of the scanners and imaging into the eyesof the user, for the user's view of the world. The rendering engine 334may be implemented by a hardware processor (such as, e.g., a centralprocessing unit or a graphics processing unit). In some embodiments, therendering engine is part of the local processing and data module 260.The rendering engine 334 can be communicatively coupled (e.g., via wiredor wireless links) to other components of the wearable system 200. Forexample, the rendering engine 334, can be coupled to the eye cameras 324via communication link 274, and be coupled to a projecting subsystem 318(which can project light into user's eyes 302, 304 via a scanned laserarrangement in a manner similar to a retinal scanning display) via thecommunication link 272. The rendering engine 334 can also be incommunication with other processing units such as, e.g., the sensor poseprocessor 332 and the image pose processor 336 via links 276 and 294respectively.

The cameras 324 (e.g., mini infrared cameras) may be utilized to trackthe eye pose to support rendering and user input. Some example eye posesmay include where the user is looking or at what depth he or she isfocusing (which may be estimated with eye vergence). The GPS 337, gyros,compass, and accelerometers 339 may be utilized to provide coarse orfast pose estimates. One or more of the cameras 316 can acquire imagesand pose, which in conjunction with data from an associated cloudcomputing resource, may be utilized to map the local environment andshare user views with others.

The example components depicted in FIG. 3 are for illustration purposesonly. Multiple sensors and other functional modules are shown togetherfor ease of illustration and description. Some embodiments may includeonly one or a subset of these sensors or modules. Further, the locationsof these components are not limited to the positions depicted in FIG. 3. Some components may be mounted to or housed within other components,such as a belt-mounted component, a hand-held component, or a helmetcomponent. As one example, the image pose processor 336, sensor poseprocessor 332, and rendering engine 334 may be positioned in a beltpackand configured to communicate with other components of the wearablesystem via wireless communication, such as ultra-wideband, Wi-Fi,Bluetooth, etc., or via wired communication. The depicted housing 230preferably is head-mountable and wearable by the user. However, somecomponents of the wearable system 200 may be worn to other portions ofthe user's body. For example, the speaker 240 may be inserted into theears of a user to provide sound to the user.

Regarding the projection of light 338 into the eyes 302, 304 of theuser, in some embodiment, the cameras 324 may be utilized to measurewhere the centers of a user's eyes are geometrically verged to, which,in general, coincides with a position of focus, or “depth of focus”, ofthe eyes. A 3-dimensional surface of all points the eyes verge to can bereferred to as the “horopter”. The focal distance may take on a finitenumber of depths, or may be infinitely varying. Light projected from thevergence distance appears to be focused to the subject eye 302, 304,while light in front of or behind the vergence distance is blurred.Examples of wearable devices and other display systems of the presentdisclosure are also described in U.S. Patent Publication No.2016/0270656, which is incorporated by reference herein in its entirety.

The human visual system is complicated and providing a realisticperception of depth is challenging. Viewers of an object may perceivethe object as being three-dimensional due to a combination of vergenceand accommodation. Vergence movements (e.g., rolling movements of thepupils toward or away from each other to converge the lines of sight ofthe eyes to fixate upon an object) of the two eyes relative to eachother are closely associated with focusing (or “accommodation”) of thelenses of the eyes. Under normal conditions, changing the focus of thelenses of the eyes, or accommodating the eyes, to change focus from oneobject to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex.” Likewise, achange in vergence will trigger a matching change in accommodation,under normal conditions. Display systems that provide a better matchbetween accommodation and vergence may form more realistic andcomfortable simulations of three-dimensional imagery.

Further spatially coherent light with a beam diameter of less than about0.7 millimeters can be correctly resolved by the human eye regardless ofwhere the eye focuses. Thus, to create an illusion of proper focaldepth, the eye vergence may be tracked with the cameras 324, and therendering engine 334 and projection subsystem 318 may be utilized torender all objects on or close to the horopter in focus, and all otherobjects at varying degrees of defocus (e.g., using intentionally-createdblurring). Preferably, the system 220 renders to the user at a framerate of about 60 frames per second or greater. As described above,preferably, the cameras 324 may be utilized for eye tracking, andsoftware may be configured to pick up not only vergence geometry butalso focus location cues to serve as user inputs. Preferably, such adisplay system is configured with brightness and contrast suitable forday or night use.

In some embodiments, the display system preferably has latency of lessthan about 20 milliseconds for visual object alignment, less than about0.1 degree of angular alignment, and about 1 arc minute of resolution,which, without being limited by theory, is believed to be approximatelythe limit of the human eye. The display system 220 may be integratedwith a localization system, which may involve GPS elements, opticaltracking, compass, accelerometers, or other data sources, to assist withposition and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (e.g., such information would facilitate the glasses toknow where they are with respect to the real world).

In some embodiments, the wearable system 200 is configured to displayone or more virtual images based on the accommodation of the user'seyes. Unlike prior 3D display approaches that force the user to focuswhere the images are being projected, in some embodiments, the wearablesystem is configured to automatically vary the focus of projectedvirtual content to allow for a more comfortable viewing of one or moreimages presented to the user. For example, if the user's eyes have acurrent focus of 1 m, the image may be projected to coincide with theuser's focus. If the user shifts focus to 3 m, the image is projected tocoincide with the new focus. Thus, rather than forcing the user to apredetermined focus, the wearable system 200 of some embodiments allowsthe user's eye to a function in a more natural manner.

Such a wearable system 200 may eliminate or reduce the incidences of eyestrain, headaches, and other physiological symptoms typically observedwith respect to virtual reality devices. To achieve this, variousembodiments of the wearable system 200 are configured to project virtualimages at varying focal distances, through one or more variable focuselements (VFEs). In one or more embodiments, 3D perception may beachieved through a multi-plane focus system that projects images atfixed focal planes away from the user. Other embodiments employ variableplane focus, wherein the focal plane is moved back and forth in thez-direction to coincide with the user's present state of focus.

In both the multi-plane focus systems and variable plane focus systems,wearable system 200 may employ eye tracking to determine a vergence ofthe user's eyes, determine the user's current focus, and project thevirtual image at the determined focus. In other embodiments, wearablesystem 200 comprises a light modulator that variably projects, through afiber scanner, or other light generating source, light beams of varyingfocus in a raster pattern across the retina. Thus, the ability of thedisplay of the wearable system 200 to project images at varying focaldistances not only eases accommodation for the user to view objects in3D, but may also be used to compensate for user ocular anomalies, asfurther described in U.S. Patent Publication No. 2016/0270656, which isincorporated by reference herein in its entirety. In some otherembodiments, a spatial light modulator may project the images to theuser through various optical components. For example, as describedfurther below, the spatial light modulator may project the images ontoone or more waveguides, which then transmit the images to the user.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2 ) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4 , as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. (Compensating lens layer 430 and thestacked waveguide assembly 480 as a whole may be configured such thatlight coming from the world 470 is conveyed to the eye 410 atsubstantially the same level of divergence (or collimation) as the lighthad when it was initially received by the stacked waveguide assembly480.) Such a configuration provides as many perceived focal planes asthere are available waveguide/lens pairings. Both the light extractingoptical elements of the waveguides and the focusing aspects of thelenses may be static (e.g., not dynamic or electro-active). In somealternative embodiments, either or both may be dynamic usingelectro-active features.

With continued reference to FIG. 4 , the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) can beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer can move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 can be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 can include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors can be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 can comprise a directionalmicrophone, as another example, which can also provide such usefuldirectional information as to where the audio source is located. Thewearable system 400 can use information from both the outward-facingimaging system 464 and the audio sensor 230 in locating a source ofspeech, or to determine an active speaker at a particular moment intime, etc. For example, the wearable system 400 can use the voicerecognition alone or in combination with a reflected image of thespeaker (e.g., as seen in a mirror) to determine the identity of thespeaker. As another example, the wearable system 400 can determine aposition of the speaker in an environment based on sound acquired fromdirectional microphones. The wearable system 400 can parse the soundcoming from the speaker's position with speech recognition algorithms todetermine the content of the speech and use voice recognition techniquesto determine the identity (e.g., name or other demographic information)of the speaker.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4 ) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

Example of an Eye Image

FIG. 5 illustrates an image of an eye 500 with eyelids 504, sclera 508(the “white” of the eye), iris 512, and pupil 516. Curve 516 a shows thepupillary boundary between the pupil 516 and the iris 512, and curve 512a shows the limbic boundary between the iris 512 and the sclera 508. Theeyelids 504 include an upper eyelid 504 a and a lower eyelid 504 b. Theeye 500 is illustrated in a natural resting pose (e.g., in which theuser's face and gaze are both oriented as they would be toward a distantobject directly ahead of the user). The natural resting pose of the eye500 can be indicated by a natural resting direction 520, which is adirection orthogonal to the surface of the eye 500 when in the naturalresting pose (e.g., directly out of the plane for the eye 500 shown inFIG. 5 ) and in this example, centered within the pupil 516.

As the eye 500 moves to look toward different objects, the eye pose willchange relative to the natural resting direction 520. The current eyepose can be determined with reference to an eye pose direction 524,which is a direction orthogonal to the surface of the eye (and centeredin within the pupil 516) but oriented toward the object at which the eyeis currently directed. With reference to an example coordinate systemshown in FIG. 5A, the pose of the eye 500 can be expressed as twoangular parameters indicating an azimuthal deflection and a zenithaldeflection of the eye pose direction 524 of the eye, both relative tothe natural resting direction 520 of the eye. For purposes ofillustration, these angular parameters can be represented as θ(azimuthal deflection, determined from a fiducial azimuth) and ϕ(zenithal deflection, sometimes also referred to as a polar deflection).In some implementations, angular roll of the eye around the eye posedirection 524 can be included in the determination of eye pose, andangular roll can be included in the following analysis. In otherimplementations, other techniques for determining the eye pose can beused, for example, a pitch, yaw, and optionally roll system.

An eye image can be obtained from a video using any appropriate process,for example, using a video processing algorithm that can extract animage from one or more sequential frames. The pose of the eye can bedetermined from the eye image using a variety of eye-trackingtechniques. For example, an eye pose can be determined by consideringthe lensing effects of the cornea on light sources that are provided.Any suitable eye tracking technique can be used for determining eye posein the eyelid shape estimation techniques described herein.

Example of an Eye Tracking System

FIG. 6 illustrates a schematic diagram of a wearable system 600 thatincludes an eye tracking system. The wearable system 600 may, in atleast some embodiments, include components located in a head-mountedunit 602 and components located in a non-head-mounted unit 604. Non-headmounted unit 604 may be, as examples, a belt-mounted component, ahand-held component, a component in a backpack, a remote component, etc.Incorporating some of the components of the wearable system 600 innon-head-mounted unit 604 may help to reduce the size, weight,complexity, and cost of the head-mounted unit 602. In someimplementations, some or all of the functionality described as beingperformed by one or more components of head-mounted unit 602 and/ornon-head mounted 604 may be provided by way of one or more componentsincluded elsewhere in the wearable system 600. For example, some or allof the functionality described below in association with a CPU 612 ofhead-mounted unit 602 may be provided by way of a CPU 616 of non-headmounted unit 604, and vice versa. In some examples, some or all of suchfunctionality may be provided by way of peripheral devices of wearablesystem 600. Furthermore, in some implementations, some or all of suchfunctionality may be provided by way of one or more cloud computingdevices or other remotely-located computing devices in a manner similarto that which has been described above with reference to FIG. 2 .

As shown in FIG. 6 , wearable system 600 can include an eye trackingsystem including a camera 324 that captures images of a user's eye 610.If desired, the eye tracking system may also include light sources 326 aand 326 b (such as light emitting diodes “LED”s). The light sources 326a and 326 b may generate glints (e.g., reflections off of the user'seyes that appear in images of the eye captured by camera 324). Thepositions of the light sources 326 a and 326 b relative to the camera324 may be known and, as a consequence, the positions of the glintswithin images captured by camera 324 may be used in tracking the user'seyes (as will be discussed in more detail below in connection with FIGS.7-11 ). In at least one embodiment, there may be one light source 326and one camera 324 associated with a single one of the user's eyes 610.In another embodiment, there may be one light source 326 and one camera324 associated with each of a user's eyes. 610. In yet otherembodiments, there may be one or more cameras 324 and one or more lightsources 326 associated with one or each of a user's eyes 610. As aspecific example, there may be two light sources 326 a and 326 b and oneor more cameras 324 associated with each of a user's eyes 610. Asanother example, there may be three or more light sources such as lightsources 326 a and 326 b and one or more cameras 324 associated with eachof a user's eyes 610.

Eye tracking module 614 may receive images from eye tracking camera(s)324 and may analyze the images to extract various pieces of information.As examples, the eye tracking module 614 may detect the user's eyeposes, a three-dimensional position of the user's eye relative to theeye tracking camera 324 (and to the head-mounted unit 602), thedirection one or both of the user's eyes 610 are focused on, the user'svergence depth (e.g., the depth from the user at which the user isfocusing on), the positions of the user's pupils, the positions of theuser's cornea and cornea sphere, the center of rotation of each of theuser's eyes, and the center of perspective of each of the user's eyes.The eye tracking module 614 may extract such information usingtechniques described below in connection with FIGS. 7-11 . As shown inFIG. 6 , eye tracking module 614 may be a software module implementedusing a CPU 612 in a head-mounted unit 602.

Data from eye tracking module 614 may be provided to other components inthe wearable system. As example, such data may be transmitted tocomponents in a non-head-mounted unit 604 such as CPU 616 includingsoftware modules for a light-field render controller 618 and aregistration observer 620.

Render controller 618 may use information from eye tracking module 614to adjust images displayed to the user by render engine 622 (e.g., arender engine that may be a software module in GPU 620 and that mayprovide images to display 220). As an example, the render controller 618may adjust images displayed to the user based on the user's center ofrotation or center of perspective. In particular, the render controller618 may use information on the user's center of perspective to simulatea render camera (e.g., to simulate collecting images from the user'sperspective) and may adjust images displayed to the user based on thesimulated render camera.

A “render camera,” which is sometimes also referred to as a “pinholeperspective camera” (or simply “perspective camera”) or “virtual pinholecamera” (or simply “virtual camera”), is a simulated camera for use inrendering virtual image content possibly from a database of objects in avirtual world. The objects may have locations and orientations relativeto the user or wearer and possibly relative to real objects in theenvironment surrounding the user or wearer. In other words, the rendercamera may represent a perspective within render space from which theuser or wearer is to view 3D virtual contents of the render space (e.g.,virtual objects). The render camera may be managed by a render engine torender virtual images based on the database of virtual objects to bepresented to said eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a pinhole camera (corresponding to the“render camera”) having a specific set of intrinsic parameters (e.g.,focal length, camera pixel size, principal point coordinates,skew/distortion parameters, etc.), and a specific set of extrinsicparameters (e.g., translational components and rotational componentsrelative to the virtual world). The virtual images are taken from theperspective of such a camera having a position and orientation of therender camera (e.g., extrinsic parameters of the render camera). Itfollows that the system may define and/or adjust intrinsic and extrinsicrender camera parameters. For example, the system may define aparticular set of extrinsic render camera parameters such that virtualimages may be rendered as if captured from the perspective of a camerahaving a specific location with respect to the user's or wearer's eye soas to provide images that appear to be from the perspective of the useror wearer. The system may later dynamically adjust extrinsic rendercamera parameters on-the-fly so as to maintain registration with saidspecific location. Similarly, intrinsic render camera parameters may bedefined and dynamically adjusted over time. In some implementations, theimages are rendered as if captured from the perspective of a camerahaving an aperture (e.g., pinhole) at a specific location with respectto the user's or wearer's eye (such as the center of perspective orcenter of rotation, or elsewhere).

In some embodiments, the system may create or dynamically repositionand/or reorient one render camera for the user's left eye, and anotherrender camera for the user's right eye, as the user's eyes arephysically separated from one another and thus consistently positionedat different locations. It follows that, in at least someimplementations, virtual content rendered from the perspective of arender camera associated with the viewer's left eye may be presented tothe user through an eyepiece on the left side of a head-mounted display(e.g., head-mounted unit 602), and that virtual content rendered fromthe perspective of a render camera associated with the user's right eyemay be presented to the user through an eyepiece on the right side ofsuch a head-mounted display. Further details discussing the creation,adjustment, and use of render cameras in rendering processes areprovided in U.S. patent application Ser. No. 15/274,823, entitled“METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN3D RECONSTRUCTION,” which is expressly incorporated herein by referencein its entirety for all purposes.

In some examples, one or more modules (or components) of the system 600(e.g., light-field render controller 618, render engine 620, etc.) maydetermine the position and orientation of the render camera withinrender space based on the position and orientation of the user's headand eyes (e.g., as determined based on head pose and eye tracking data,respectively). That is, the system 600 may effectively map the positionand orientation of the user's head and eyes to particular locations andangular positions within a 3D virtual environment, place and orientrender cameras at the particular locations and angular positions withinthe 3D virtual environment, and render virtual content for the user asit would be captured by the render camera. Further details discussingreal world to virtual world mapping processes are provided in U.S.patent application Ser. No. 15/296,869, entitled “SELECTING VIRTUALOBJECTS IN A THREE-DIMENSIONAL SPACE,” which is expressly incorporatedherein by reference in its entirety for all purposes. As an example, therender controller 618 may adjust the depths at which images aredisplayed by selecting which depth plane (or depth planes) are utilizedat any given time to display the images. In some implementations, such adepth plane switch may be carried out through an adjustment of one ormore intrinsic render camera parameters. For example, the light-fieldrender controller 618 may adjust the focal lengths of render cameraswhen executing a depth plane switch or adjustment. As described infurther detail below, depth planes may be switched based on the user'sdetermined vergence or fixation depth.

Registration observer 620 may use information from eye tracking module614 to identify whether the head-mounted unit 602 is properly positionedon a user's head. As an example, the eye tracking module 614 may provideeye location information, such as the positions of the centers ofrotation of the user's eyes, indicative of the three-dimensionalposition of the user's eyes relative to camera 324 and head-mounted unit602 and the eye tracking module 614 may use the location information todetermine if display 220 is properly aligned in the user's field ofview, or if the head-mounted unit 602 (or headset) has slipped or isotherwise misaligned with the user's eyes. As examples, the registrationobserver 620 may be able to determine if the head-mounted unit 602 hasslipped down the user's nose bridge, thus moving display 220 away anddown from the user's eyes (which may be undesirable), if thehead-mounted unit 602 has been moved up the user's nose bridge, thusmoving display 220 closer and up from the user's eyes, if thehead-mounted unit 602 has been shifted left or right relative the user'snose bridge, if the head-mounted unit 602 has been lifted above theuser's nose bridge, or if the head-mounted unit 602 has been moved inthese or other ways away from a desired position or range of positions.In general, registration observer 620 may be able to determine ifhead-mounted unit 602, in general, and displays 220, in particular, areproperly positioned in front of the user's eyes. In other words, theregistration observer 620 may determine if a left display in displaysystem 220 is appropriately aligned with the user's left eye and a rightdisplay in display system 220 is appropriately aligned with the user'sright eye. The registration observer 620 may determine if thehead-mounted unit 602 is properly positioned by determining if thehead-mounted unit 602 is positioned and oriented within a desired rangeof positions and/or orientations relative to the user's eyes.

In at least some embodiments, registration observer 620 may generateuser feedback in the form of alerts, messages, or other content. Suchfeedback may be provided to the user to inform the user of anymisalignment of the head-mounted unit 602, along with optional feedbackon how to correct the misalignment (such as a suggestion to adjust thehead-mounted unit 602 in a particular manner).

Example registration observation and feedback techniques, which may beutilized by registration observer 620, are described in U.S. patentapplication Ser. No. 15/717,747, filed Sep. 27, 2017 and U.S.Provisional Patent Application No. 62/644,321, filed Mar. 16, 2018, bothof which are incorporated by reference herein in their entirety.

Example of an Eye Tracking Module

A detailed block diagram of an example eye tracking module 614 is shownin FIG. 7A. As shown in FIG. 7A, eye tracking module 614 may include avariety of different submodules, may provide a variety of differentoutputs, and may utilize a variety of available data in tracking theuser's eyes. As examples, eye tracking module 614 may utilize availabledata including eye tracking extrinsics and intrinsics, such as thegeometric arrangements of the eye tracking camera 324 relative to thelight sources 326 and the head-mounted-unit 602; assumed eye dimensions704 such as a typical distance of approximately 4.7 mm between a user'scenter of cornea curvature and the average center of rotation of theuser's eye or typical distances between a user's center of rotation andcenter of perspective; and per-user calibration data 706 such as aparticular user's interpupillary distance. Additional examples ofextrinsics, intrinsics, and other information that may be employed bythe eye tracking module 614 are described in U.S. patent applicationSer. No. 15/497,726, filed Apr. 26, 2017, which is incorporated byreference herein in its entirety.

Image preprocessing module 710 may receive images from an eye camerasuch as eye camera 324 and may perform one or more preprocessing (e.g.,conditioning) operations on the received images. As examples, imagepreprocessing module 710 may apply a Gaussian blur to the images, maydown sample the images to a lower resolution, may applying an unsharpmask, may apply an edge sharpening algorithm, or may apply othersuitable filters that assist with the later detection, localization, andlabelling of glints, a pupil, or other features in the images from eyecamera 324. The image preprocessing module 710 may apply a low-passfilter or a morphological filter such as an open filter, which canremove high-frequency noise such as from the pupillary boundary 516 a(see FIG. 5 ), thereby removing noise that can hinder pupil and glintdetermination. The image preprocessing module 710 may outputpreprocessed images to the pupil identification module 712 and to theglint detection and labeling module 714.

Pupil identification module 712 may receive preprocessed images from theimage preprocessing module 710 and may identify regions of those imagesthat include the user's pupil. The pupil identification module 712 may,in some embodiments, determine the coordinates of the position, orcoordinates, of the center, or centroid, of the user's pupil in the eyetracking images from camera 324. In at least some embodiments, pupilidentification module 712 may identify contours in eye tracking images(e.g., contours of pupil iris boundary), identify contour moments (e.g.,centers of mass), apply a starburst pupil detection and/or a canny edgedetection algorithm, reject outliers based on intensity values, identifysub-pixel boundary points, correct for eye-camera distortion (e.g.,distortion in images captured by eye camera 324), apply a random sampleconsensus (RANSAC) iterative algorithm to fit an ellipse to boundariesin the eye tracking images, apply a tracking filter to the images, andidentify sub-pixel image coordinates of the user's pupil centroid. Thepupil identification module 712 may output pupil identification data,which may indicate which regions of the preprocessing images module 712identified as showing the user's pupil, to glint detection and labelingmodule 714. The pupil identification module 712 may provide the 2Dcoordinates of the user's pupil (e.g., the 2D coordinates of thecentroid of the user's pupil) within each eye tracking image to glintdetection module 714. In at least some embodiments, pupil identificationmodule 712 may also provide pupil identification data of the same sortto coordinate system normalization module 718.

Pupil detection techniques, which may be utilized by pupilidentification module 712, are described in U.S. Patent Publication No.2017/0053165, published Feb. 23, 2017 and in U.S. Patent Publication No.2017/0053166, published Feb. 23, 2017, each of which is incorporated byreference herein in its entirety.

Glint detection and labeling module 714 may receive preprocessed imagesfrom module 710 and pupil identification data from module 712. Glintdetection module 714 may use this data to detect and/or identify glints(e.g., reflections off of the user's eye of the light from light sources326) within regions of the preprocessed images that show the user'spupil. As an example, the glint detection module 714 may search forbright regions within the eye tracking image, sometimes referred toherein as “blobs” or local intensity maxima, that are in the vicinity ofthe user's pupil. In at least some embodiments, the glint detectionmodule 714 may rescale (e.g., enlarge) the pupil ellipse to encompassadditional glints. The glint detection module 714 may filter glints bysize and/or by intensity. The glint detection module 714 may alsodetermine the 2D positions of each of the glints within the eye trackingimage.

In at least some examples, the glint detection module 714 may determinethe 2D positions of the glints relative to the user's pupil, which mayalso be referred to as the pupil-glint vectors. Glint detection andlabeling module 714 may label the glints and output the preprocessingimages with labeled glints to the 3D cornea center estimation module716. Glint detection and labeling module 714 may also pass along datasuch as preprocessed images from module 710 and pupil identificationdata from module 712. In some implementations, the glint detection andlabeling module 714 may determine which light source (e.g., from among aplurality of light sources of the system including infrared lightsources 326 a and 326 b) produced each identified glint. In theseexamples, the glint detection and labeling module 714 may label theglints with information identifying the associated light source andoutput the preprocessing images with labeled glints to the 3D corneacenter estimation module 716.

Pupil and glint detection, as performed by modules such as modules 712and 714, can use any suitable techniques. As examples, edge detectioncan be applied to the eye image to identify glints and pupils. Edgedetection can be applied by various edge detectors, edge detectionalgorithms, or filters. For example, a Canny Edge detector can beapplied to the image to detect edges such as in lines of the image.Edges may include points located along a line that correspond to thelocal maximum derivative. For example, the pupillary boundary 516 a (seeFIG. 5 ) can be located using a Canny edge detector. With the locationof the pupil determined, various image processing techniques can be usedto detect the “pose” of the pupil 116. Determining an eye pose of an eyeimage can also be referred to as detecting an eye pose of the eye image.The pose can also be referred to as the gaze, pointing direction, or theorientation of the eye. For example, the pupil may be looking leftwardstowards an object, and the pose of the pupil could be classified as aleftwards pose. Other methods can be used to detect the location of thepupil or glints. For example, a concentric ring can be located in an eyeimage using a Canny Edge detector. As another example, anintegro-differential operator can be used to find the pupillary orlimbus boundaries of the iris. For example, the Daugmanintegro-differential operator, the Hough transform, or other irissegmentation techniques can be used to return a curve that estimates theboundary of the pupil or the iris.

3D cornea center estimation module 716 may receive preprocessed imagesincluding detected glint data and pupil identification data from modules710, 712, 714. 3D cornea center estimation module 716 may use these datato estimate the 3D position of the user's cornea. In some embodiments,the 3D cornea center estimation module 716 may estimate the 3D positionof an eye's center of cornea curvature or a user's corneal sphere, e.g.,the center of an imaginary sphere having a surface portion generallycoextensive with the user's cornea. The 3D cornea center estimationmodule 716 may provide data indicating the estimated 3D coordinates ofthe corneal sphere and/or user's cornea to the coordinate systemnormalization module 718, the optical axis determination module 722,and/or the light-field render controller 618. Further details of theoperation of the 3D cornea center estimation module 716 are providedherein in connection with FIGS. 8A-8E. Techniques for estimating thepositions of eye features such as a cornea or corneal sphere, which maybe utilized by 3D cornea center estimation module 716 and other modulesin the wearable systems of the present disclosure are discussed in U.S.patent application Ser. No. 15/497,726, filed Apr. 26, 2017, which isincorporated by reference herein in its entirety.

Coordinate system normalization module 718 may optionally (as indicatedby its dashed outline) be included in eye tracking module 614.Coordinate system normalization module 718 may receive data indicatingthe estimated 3D coordinates of the center of the user's cornea (and/orthe center of the user's corneal sphere) from the 3D cornea centerestimation module 716 and may also receive data from other modules.Coordinate system normalization module 718 may normalize the eye cameracoordinate system, which may help to compensate for slippages of thewearable device (e.g., slippages of the head-mounted component from itsnormal resting position on the user's head, which may be identified byregistration observer 620). Coordinate system normalization module 718may rotate the coordinate system to align the z-axis (e.g., the vergencedepth axis) of the coordinate system with the cornea center (e.g., asindicated by the 3D cornea center estimation module 716) and maytranslate the camera center (e.g., the origin of the coordinate system)to a predetermined distance away from the cornea center such as 30 mm(e.g., module 718 may enlarge or shrink the eye tracking image dependingon whether the eye camera 324 was determined to be nearer or furtherthan the predetermined distance). With this normalization process, theeye tracking module 614 may be able to establish a consistentorientation and distance in the eye tracking data, relativelyindependent of variations of headset positioning on the user's head.Coordinate system normalization module 718 may provide 3D coordinates ofthe center of the cornea (and/or corneal sphere), pupil identificationdata, and preprocessed eye tracking images to the 3D pupil centerlocator module 720. Further details of the operation of the coordinatesystem normalization module 718 are provided herein in connection withFIGS. 9A-9C.

3D pupil center locator module 720 may receive data, in the normalizedor the unnormalized coordinate system, including the 3D coordinates ofthe center of the user's cornea (and/or corneal sphere), pupil locationdata, and preprocessed eye tracking images. 3D pupil center locatormodule 720 may analyze such data to determine the 3D coordinates of thecenter of the user's pupil in the normalized or unnormalized eye cameracoordinate system. The 3D pupil center locator module 720 may determinethe location of the user's pupil in three-dimensions based on the 2Dposition of the pupil centroid (as determined by module 712), the 3Dposition of the cornea center (as determined by module 716), assumed eyedimensions 704 such as the size of the a typical user's corneal sphereand the typical distance from the cornea center to the pupil center, andoptical properties of eyes such as the index of refraction of the cornea(relative to the index of refraction of air) or any combination ofthese. Further details of the operation of the 3D pupil center locatormodule 720 are provided herein in connection with FIGS. 9D-9G.Techniques for estimating the positions of eye features such as a pupil,which may be utilized by 3D pupil center locator module 720 and othermodules in the wearable systems of the present disclosure are discussedin U.S. patent application Ser. No. 15/497,726, filed Apr. 26, 2017,which is incorporated by reference herein in its entirety.

Optical axis determination module 722 may receive data from modules 716and 720 indicating the 3D coordinates of the center of the user's corneaand the user's pupil. Based on such data, the optical axis determinationmodule 722 may identify a vector from the position of the cornea center(e.g., from the center of the corneal sphere) to the center of theuser's pupil, which may define the optical axis of the user's eye.Optical axis determination module 722 may provide outputs specifying theuser's optical axis to modules 724, 728, 730, and 732, as examples.

Center of rotation (CoR) estimation module 724 may receive data frommodule 722 including parameters of the optical axis of the user's eye(e.g., data indicating the direction of the optical axis in a coordinatesystem with a known relation to the head-mounted unit 602). For example,CoR estimation module 724 may estimate the center of rotation of auser's eye. The center of rotation may indicate a point around which theuser's eye rotates when the user eye rotates left, right, up, and/ordown. While eyes may not rotate perfectly around a singular point,assuming a singular point may be sufficient. In at least someembodiments, CoR estimation module 724 may estimate an eye's center ofrotation by moving from the center of the pupil (identified by module720) or the center of curvature of the cornea (as identified by module716) toward the retina along the optical axis (identified by module 722)a particular distance. This particular distance may be an assumed eyedimension 704. As one example, the particular distance between thecenter of curvature of the cornea and the CoR may be approximately 4.7mm. This distance may be varied for a particular user based on anyrelevant data including the user's age, sex, vision prescription, otherrelevant characteristics, etc. Additional discussion of the value of 4.7mm as an estimate for the distance between the center of curvature ofthe cornea and the CoR is provided in Appendix (Part III), which formspart of this application.

In at least some embodiments, the CoR estimation module 724 may refineits estimate of the center of rotation of each of the user's eyes overtime. As an example, as time passes, the user will eventually rotatetheir eyes (to look somewhere else, at something closer, further, orsometime left, right, up, or down) causing a shift in the optical axisof each of their eyes. CoR estimation module 724 may then analyze two(or more) optical axes identified by module 722 and locate the 3D pointof intersection of those optical axes. The CoR estimation module 724 maythen determine the center of rotation lies at that 3D point ofintersection. Such a technique may provide for an estimate of the centerof rotation, with an accuracy that improves over time.

Various techniques may be employed to increase the accuracy of the CoRestimation module 724 and the determined CoR positions of the left andright eyes. As an example, the CoR estimation module 724 may estimatethe CoR by finding the average point of intersection of optical axesdetermined for various different eye poses over time. As additionalexamples, module 724 may filter or average estimated CoR positions overtime, may calculate a moving average of estimated CoR positions overtime, and/or may apply a Kalman filter and known dynamics of the eyesand eye tracking system to estimate the CoR positions over time. In someimplementations, a least-squares approach may be taken to determine oneor more points of intersection of optical axes. In such implementations,the system may, at a given point in time, identify a location at whichthe sum of the squared distances to a given set of optical axes isreduced or minimized as the point of optical axes intersection. As aspecific example, module 724 may calculate a weighted average ofdetermined points of optical axes intersection and assumed CoR positions(such as 4.7 mm from an eye's center of cornea curvature), such that thedetermined CoR may slowly drift from an assumed CoR position (e.g., 4.7mm behind an eye's center of cornea curvature) to a slightly differentlocation within the user's eye over time as eye tracking data for theuser is obtain and thereby enables per-user refinement of the CoRposition.

Under ideal conditions, the 3D position of the true CoR of a user's eyerelative to the HMD should change a negligible or minimal amount overtime as the user moves their eye (e.g., as the user's eye rotates aroundits center of rotation). In other words, for a given set of eyemovements, the 3D position of the true CoR of the user's eye (relativeto the HMD) should hypothetically vary less over time than any otherpoint along the optical axis of the user's eye. As such, it follows thatthe further away a point along the optical axis is from the true CoR ofthe user's eye, the more variation or variance its 3D position willexhibit over time as the user moves their eye. In some embodiments, theCoR estimation module 724 and/or other submodules of eye tracking module614 may make use of this statistical relationship to improve CoRestimation accuracy. In such embodiments, the CoR estimation module 724and/or other submodules of eye tracking module 614 may refine theirestimates of the CoR 3D position over time by identifying variations ofits CoR estimates having a low variation (e.g., low variance or standarddeviation).

As a first example and in embodiments where the CoR estimation module724 estimates CoR based on intersection of multiple different opticalaxes (each associated with the user looking in a different direction),the CoR estimation module 724 may make use of this statisticalrelationship (that the true CoR should have a low variance) byintroducing common offsets to the direction of each of the optical axes(e.g., shifting each axis by some uniform amount) and determining if theoffset optical axes intersect with each other in an intersection pointhaving a low variation, e.g., low variance or standard deviation. Thismay correct for minor systemic errors in calculating the directions ofthe optical axes and help to refine the estimated position of the CoR tobe closer to the true CoR.

As a second example and in embodiments where the CoR estimation module724 estimates CoR by moving along an optical axis (or other axis) by aparticular distance (e.g., such as the distance between the center ofcurvature of the cornea and the CoR), the system may vary, optimize,tune, or otherwise adjust the particular distance between the center ofcurvature of the cornea and the CoR over time (for example, for a largegroup of images of the eye captured at different times) in a manner soas to reduce or minimize the variation, for example, variance and/orstandard deviation of the estimated CoR position. For example, if theCoR estimation module 724 initially uses a particular distance value of4.7 mm (from the center of curvature of the cornea and along the opticalaxis) to obtain CoR position estimates, but the true CoR of a givenuser's eye may be positioned 4.9 mm behind the eye's center of corneacurvature (along the optical axis), then an initial set of CoR positionestimates obtained by the CoR estimation module 724 may exhibit arelatively high amount of variation, e.g., variance or standarddeviation. In response to detecting such a relatively high amount ofvariation (e.g., variance or standard deviation), the CoR estimationmodule 724 may look for and identify one or more points along theoptical axis having a lower amount of variation (e.g., variance orstandard deviation), may identify the 4.9 mm distance as having thelowest variation (e.g., variance or standard deviation), and may thusadjust the particular distance value utilized to 4.9 mm.

The CoR estimation module 724 may look for alternative CoR estimationshaving lower variation (e.g., variance and/or standard deviation) inresponse to detecting that a current CoR estimate has a relatively highamount of variation (e.g., variance or standard deviation) or may lookfor alternative CoR estimations having lower variation (e.g. variance orstandard deviation) as a matter of course after obtaining initial CoRestimates. In some examples, such an optimization/adjustment can happengradually over time, while in other examples, such anoptimization/adjustment can be made during an initial user calibrationsession. In examples where such a procedure is conducted during acalibration procedure, the CoR estimation module 724 may not initiallysubscribe/adhere to any assumed particular distance, but may rathercollect a set of eye tracking data over time, perform statisticalanalysis on the set of eye tracking data, and determine the particulardistance value yielding CoR position estimates with the least possibleamount (e.g., global minima) of variation (e.g. variance or standarddeviation) based on the statistical analysis.

Additional discussion of the statistical relationship described above(e.g., that the true CoR should have low variance or standarddeviation), as well as the significance of taking into account cornealrefraction in determining pupil position, is provided in Appendix (PartIII), which forms part of this application.

Interpupillary distance (IPD) estimation module 726 may receive datafrom CoR estimation module 724 indicating the estimated 3D positions ofthe centers of rotation of the user's left and right eyes. IPDestimation module 726 may then estimate a user's IPD by measuring the 3Ddistance between the centers of rotation of the user's left and righteyes. In general, the distance between the estimated CoR of the user'sleft eye and the estimated CoR of the user's right eye may be roughlyequal to the distance between the centers of a user's pupils, when theuser is looking at optical infinity (e.g., the optical axes of theuser's eyes are substantially parallel to one another), which is thetypical definition of interpupillary distance (IPD). A user's IPD may beused by various components and modules in the wearable system. Asexample, a user's IPD may be provided to registration observer 620 andused in assessing how well the wearable device is aligned with theuser's eyes (e.g., whether the left and right display lenses areproperly spaced in accordance with the user's IPD). As another example,a user's IPD may be provided to vergence depth estimation module 728 andbe used in determining a user's vergence depth. Module 726 may employvarious techniques, such as those discussed in connection with CoRestimation module 724, to increase the accuracy of the estimated IPD. Asexamples, IPD estimation module 724 may apply filtering, averaging overtime, weighted averaging including assumed IPD distances, Kalmanfilters, etc. as part of estimating a user's IPD in an accurate manner.

Vergence depth estimation module 728 may receive data from variousmodules and submodules in the eye tracking module 614 (as shown inconnection with FIG. 7A). In particular, vergence depth estimationmodule 728 may employ data indicating estimated 3D positions of pupilcenters (e.g., as provided by module 720 described above), one or moredetermined parameters of optical axes (e.g., as provided by module 722described above), estimated 3D positions of centers of rotation (e.g.,as provided by module 724 described above), estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as provided by module 726 described above), and/or oneor more determined parameters of optical and/or visual axes (e.g., asprovided by module 722 and/or module 730 described below). Vergencedepth estimation module 728 may detect or otherwise obtain a measure ofa user's vergence depth, which may be the distance from the user atwhich the user's eyes are focused. As examples, when the user is lookingat an object three feet in front of them, the user's left and right eyeshave a vergence depth of three feet; and, while when the user is lookingat a distant landscape (e.g., the optical axes of the user's eyes aresubstantially parallel to one another such that the distance between thecenters of the user's pupils may be roughly equal to the distancebetween the centers of rotation of the user's left and right eyes), theuser's left and right eyes have a vergence depth of infinity. In someimplementations, the vergence depth estimation module 728 may utilizedata indicating the estimated centers of the user's pupils (e.g., asprovided by module 720) to determine the 3D distance between theestimated centers of the user's pupils. The vergence depth estimationmodule 728 may obtain a measure of vergence depth by comparing such adetermined 3D distance between pupil centers to estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as indicated by module 726 described above). Inaddition to the 3D distance between pupil centers and estimated IPD, thevergence depth estimation module 728 may utilize known, assumed,estimated, and/or determined geometries to calculate vergence depth. Asan example, module 728 may combine 3D distance between pupil centers,estimated IPD, and 3D CoR positions in a trigonometric calculation toestimate (e.g., determine) a user's vergence depth. Indeed, anevaluation of such a determined 3D distance between pupil centersagainst estimated IPD may serve to indicate a measure of the user'scurrent vergence depth relative to optical infinity. In some examples,the vergence depth estimation module 728 may simply receive or accessdata indicating an estimated 3D distance between the estimated centersof the user's pupils for purposes of obtaining such a measure ofvergence depth. In some embodiments, the vergence depth estimationmodule 728 may estimate vergence depth by comparing a user's left andright optical axis. In particular, vergence depth estimation module 728may estimate vergence depth by locating the distance from a user atwhich the user's left and right optical axes intersect (or whereprojections of the user's left and right optical axes on a plane such asa horizontal plane intersect). Module 728 may utilize a user's IPD inthis calculation, by setting the zero depth to be the depth at which theuser's left and right optical axes are separated by the user's IPD. Inat least some embodiments, vergence depth estimation module 728 maydetermine vergence depth by triangulating eye tracking data togetherwith known or derived spatial relationships.

In some embodiments, vergence depth estimation module 728 may estimate auser's vergence depth based on the intersection of the user's visualaxes (instead of their optical axes), which may provide a more accurateindication of the distance at which the user is focused on. In at leastsome embodiments, eye tracking module 614 may include optical to visualaxis mapping module 730. As discussed in further detail in connectionwith FIG. 10 , a user's optical and visual axis are generally notaligned. A visual axis is the axis along which a person is looking,while an optical axis is defined by the center of that person's lens andpupil, and may go through the center of the person's retina. Inparticular, a user's visual axis is generally defined by the location ofthe user's fovea, which may be offset from the center of a user'sretina, thereby resulting in different optical and visual axis. In atleast some of these embodiments, eye tracking module 614 may includeoptical to visual axis mapping module 730. Optical to visual axismapping module 730 may correct for the differences between a user'soptical and visual axis and provide information on the user's visualaxis to other components in the wearable system, such as vergence depthestimation module 728 and light-field render controller 618. In someexamples, module 730 may use assumed eye dimensions 704 including atypical offset of approximately 5.2° inwards (nasally, towards a user'snose) between an optical axis and a visual axis. In other words, module730 may shift a user's left optical axis (nasally) rightwards by 5.2°towards the nose and a user's right optical axis (nasally) leftwards by5.2° towards the nose in order to estimate the directions of the user'sleft and right optical axes. In other examples, module 730 may utilizeper-user calibration data 706 in mapping optical axes (e.g., asindicated by module 722 described above) to visual axes. As additionalexamples, module 730 may shift a user's optical axes nasally by between4.0° and 6.5°, by between 4.5° and 6.0°, by between 5.0° and 5.4°, etc.,or any ranges formed by any of these values. In some arrangements, themodule 730 may apply a shift based at least in part upon characteristicsof a particular user such as their age, sex, vision prescription, orother relevant characteristics and/or may apply a shift based at leastin part upon a calibration process for a particular user (e.g., todetermine a particular user's optical-visual axis offset). In at leastsome embodiments, module 730 may also shift the origins of the left andright optical axes to correspond with the user's CoP (as determined bymodule 732) instead of the user's CoR.

Optional center of perspective (CoP) estimation module 732, whenprovided, may estimate the location of the user's left and right centersof perspective (CoP). A CoP may be a useful location for the wearablesystem and, in at least some embodiments, is a position just in front ofa pupil. In at least some embodiments, CoP estimation module 732 mayestimate the locations of a user's left and right centers of perspectivebased on the 3D location of a user's pupil center, the 3D location of auser's center of cornea curvature, or such suitable data or anycombination thereof. As an example, a user's CoP may be approximately5.01 mm in front of the center of cornea curvature (e.g., 5.01 mm fromthe corneal sphere center in a direction that is towards the eye'scornea and that is along the optical axis) and may be approximately 2.97mm behind the outer surface of a user's cornea, along the optical orvisual axis. A user's center of perspective may be just in front of thecenter of their pupil. As examples, a user's CoP may be less thanapproximately 2.0 mm from the user's pupil, less than approximately 1.0mm from the user's pupil, or less than approximately 0.5 mm from theuser's pupil or any ranges between any of these values. As anotherexample, the center of perspective may correspond to a location withinthe anterior chamber of the eye. As other examples, the CoP may bebetween 1.0 mm and 2.0 mm, about 1.0 mm, between 0.25 mm and 1.0 mm,between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

The center of perspective described herein (as a potentially desirableposition for a pinhole of a render camera and an anatomical position ina user's eye) may be a position that serves to reduce and/or eliminateundesired parallax shifts. In particular, the optical system of a user'seye is very roughly equivalent to theoretical system formed by a pinholein front of a lens, projecting onto a screen, with the pinhole, lens,and screen roughly corresponding to a user's pupil/iris, lens, andretina, respectively. Moreover, it may be desirable for there to belittle or no parallax shift when two point light sources (or objects) atdifferent distances from the user's eye are rigidly rotated about theopening of the pinhole (e.g., rotated along radii of curvature equal totheir respective distance from the opening of the pinhole). Thus, itwould seem that the CoP should be located at the center of the pupil ofan eye (and such a CoP may be used in some embodiments). However, thehuman eye includes, in addition to the lens and pinhole of the pupil, acornea that imparts additional optical power to light propagating towardthe retina). Thus, the anatomical equivalent of the pinhole in thetheoretical system described in this paragraph may be a region of theuser's eye positioned between the outer surface of the cornea of theuser's eye and the center of the pupil or iris of the user's eye. Forinstance, the anatomical equivalent of the pinhole may correspond to aregion within the anterior chamber of a user's eye. For various reasonsdiscussed herein, it may be desired to set the CoP to such a positionwithin the anterior chamber of the user's eye. The derivation andsignificance of the CoP are described in more detail below, with respectto FIGS. 22-24B.

As discussed above, eye tracking module 614 may provide data, such asestimated 3D positions of left and right eye centers of rotation (CoR),vergence depth, left and right eye optical axis, 3D positions of auser's eye, 3D positions of a user's left and right centers of corneacurvature, 3D positions of a user's left and right pupil centers, 3Dpositions of a user's left and right center of perspective, a user'sIPD, etc., to other components, such as light-field render controller618 and registration observer 620, in the wearable system. Eye trackingmodule 614 may also include other submodules that detect and generatedata associated with other aspects of a user's eye. As examples, eyetracking module 614 may include a blink detection module that provides aflag or other alert whenever a user blinks and a saccade detectionmodule that provides a flag or other alert whenever a user's eyesaccades (e.g., quickly shifts focus to another point).

Example of a Render Controller

A detailed block diagram of an example light-field render controller 618is shown in FIG. 7B. As shown in FIGS. 6 and 7B, render controller 618may receive eye tracking information from eye tracking module 614 andmay provide outputs to render engine 622, which may generate images tobe displayed for viewing by a user of the wearable system. As examples,render controller 618 may receive a vergence depth, left and right eyecenters of rotation (and/or centers of perspective), and other eye datasuch as blink data, saccade data, etc.

Depth plane selection module 750 may receive vergence depth informationand other eye data and, based on such data, may cause render engine 622to convey content to a user with a particular depth plane (e.g., at aparticular accommodation or focal distance). As discussed in connectionwith FIG. 4 , a wearable system may include a plurality of discretedepth planes formed by a plurality of waveguides, each conveying imageinformation with a varying level of wavefront curvature. In someembodiments, a wearable system may include one or more variable depthplanes, such as an optical element that conveys image information with alevel of wavefront curvature that varies over time. In these and otherembodiments, depth plane selection module 750 may cause render engine622 to convey content to a user at a selected depth (e.g., cause renderengine 622 to direct display 220 to switch depth planes), based in partof the user's vergence depth. In at least some embodiments, depth planeselection module 750 and render engine 622 may render content atdifferent depths and also generate and/or provide depth plane selectiondata to display hardware such as display 220. Display hardware such asdisplay 220 may perform an electrical depth plane switching in responseto depth plane selection data (which may be control signals) generatedby and/or provided by modules such as depth plane selection module 750and render engine 622.

In general, it may be desirable for depth plane selection module 750 toselect a depth plane matching the user's current vergence depth, suchthat the user is provided with accurate accommodation cues. However, itmay also be desirable to switch depth planes in a discreet andunobtrusive manner. As examples, it may be desirable to avoid excessiveswitching between depth planes and/or it may be desire to switch depthplanes at a time when the user is less likely to notice the switch, suchas during a blink or eye saccade.

Hysteresis band crossing detection module 752 may help to avoidexcessive switching between depth planes, particularly when a user'svergence depth fluctuates at the midpoint or transition point betweentwo depth planes. In particular, module 752 may cause depth planeselection module 750 to exhibit hysteresis in its selection of depthplanes. As an example, modules 752 may cause depth plane selectionmodule 750 to switch from a first farther depth plane to a second closerdepth plane only after a user's vergence depth passes a first threshold.Similarly, module 752 may cause depth plane selection module 750 (whichmay in turn direct displays such as display 220) to switch to the firstfarther depth plane only after the user's vergence depth passes a secondthreshold that is farther from the user than the first threshold. In theoverlapping region between the first and second thresholds, module 750may cause depth plane selection module 750 to maintain whichever depthplane is currently select as the selected depth plane, thus avoidingexcessive switching between depth planes.

Ocular event detection module 750 may receive other eye data from theeye tracking module 614 of FIG. 7A and may cause depth plane selectionmodule 750 to delay some depth plane switches until an ocular eventoccurs. As an example, ocular event detection module 750 may cause depthplane selection module 750 to delay a planned depth plane switch until auser blink is detected; may receive data from a blink detectioncomponent in eye tracking module 614 that indicates when the user iscurrently blinking; and, in response, may cause depth plane selectionmodule 750 to execute the planned depth plane switch during the blinkevent (such by causing module 750 to direct display 220 to execute thedepth plane switch during the blink event). In at least someembodiments, the wearable system may be able to shift content onto a newdepth plane during a blink event such that the user is unlikely toperceive the shift. As another example, ocular event detection module750 may delay planned depth plane switches until an eye saccade isdetected. As discussed in connection with eye blinks, such as anarrangement may facilitate the discretely shifting of depth planes.

If desired, depth plane selection module 750 may delay planned depthplane switches only for a limited period of time before executing thedepth plane switch, even in the absence of an ocular event. Similarly,depth plane selection module 750 may execute a depth plane switch whenthe user's vergence depth is substantially outside of acurrently-selected depth plane (e.g., when the user's vergence depth hasexceeded a predetermined threshold beyond the regular threshold for adepth plane switch), even in the absence of an ocular event. Thesearrangements may help ensure that ocular event detection module 754 doesnot indefinitely delay depth plane switches and does not delay depthplane switches when a large accommodation error is present. Furtherdetails of the operation of depth plane selection module 750, and howthe module may time depth plane switches, are provided herein inconnection with FIG. 12 .

Render camera controller 758 may provide information to render engine622 indicating where the user's left and right eyes are. Render engine622 may then generate content by simulating cameras at the positions ofthe user's left and right eyes and generating content based on theperspectives of the simulated cameras. As discussed above, the rendercamera is a simulated camera for use in rendering virtual image contentpossibly from a database of objects in a virtual world. The objects mayhave locations and orientations relative to the user or wearer andpossibly relative to real objects in the environment surrounding theuser or wearer. The render camera may be included in a render engine torender virtual images based on the database of virtual objects to bepresented to said eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a camera (corresponding to the “rendercamera”) having an aperture, lens, and detector viewing the objects inthe virtual world. The virtual images are taken from the perspective ofsuch a camera having a position of the “render camera.” For example, thevirtual images may be rendered as if captured from the perspective of acamera having a specific location with respect to the user's or wearer'seye so as to provide images that appear to be from the perspective ofthe user or wearer. In some implementations, the images are rendered asif captured from the perspective of a camera having an aperture at aspecific location with respect to the user's or wearer's eye (such asthe center of perspective or center of rotation as discussed herein, orelsewhere).

Render camera controller 758 may determine the positions of the left andright cameras based on the left and right eye centers of rotation (CoR),determined by CoR estimation module 724, and/or based on the left andright eye centers of perspective (CoP), determined by CoP estimationmodule 732. In some embodiments, render camera controller 758 may switchbetween the CoR and CoP locations based on various factors. As examples,the render camera controller 758 may, in various modes, register therender camera to the CoR locations at all times, register the rendercamera to the CoP locations at all times, toggle or discretely switchbetween registering the render camera to the CoR locations andregistering the render camera to the CoP locations over time based onvarious factors, or dynamically register the render camera to any of arange of different positions along the optical (or visual) axis betweenthe CoR and CoP locations over time based on various factors. The CoRand CoP positions may optionally pass through smoothing filter 756 (inany of the aforementioned modes for render camera positioning) which mayaverage the CoR and CoP locations over time to reduce noise in thesepositions and prevent jitter in the render simulated render cameras.

In at least some embodiments, the render camera may be simulated as apinhole camera with the pinhole disposed at the position of theestimated CoR or CoP identified by eye tracking module 614. As the CoPis offset from the CoR, the location of the render camera and itspinhole both shift as the user's eye rotates, whenever the rendercamera's position is based on a user's CoP (see, e.g., how the rendercamera linearly translates with the eye rotation as shown in FIGS. 16Aand 16B). In contrast, whenever the render camera's position is based ona user's CoR, the location of the render camera's pinhole does not movewith eye rotations, although the render camera (which is behind thepinhole) may, in some embodiments, move with eye rotation. In otherembodiments where the render camera's position is based on a user's CoR,the render camera may not move (e.g., rotate) with a user's eye (see,e.g., how the render camera does not move, or linearly translate, withthe eye rotation depicted in FIGS. 17A and 17B).

Example of Locating a User's Cornea with an Eye Tracking System

FIG. 8A is a schematic diagram of an eye showing the eye's cornealsphere. As shown in FIG. 8A, a user's eye 810 may have a cornea 812, apupil 822, and a lens 820. The cornea 812 may have an approximatelyspherical shape, shown by corneal sphere 814. Corneal sphere 814 mayhave a center point 816, also referred to as a corneal center, and aradius 818. The semispherical cornea of a user's eye may curve aroundthe corneal center 816.

FIGS. 8B-8E illustrate an example of locating a user's corneal center816 using 3D cornea center estimation module 716 and eye tracking module614.

As shown in FIG. 8B, 3D cornea center estimation module 716 may receivean eye tracking image 852 that includes a corneal glint 854. The 3Dcornea center estimation module 716 may then simulate, in an eye cameracoordinate system 850, the known 3D positions of the eye camera 324 andlight source 326 (which may be based on data in eye tracking extrinsics& intrinsics database 702, assumed eye dimensions database 704, and/orper-user calibration data 706) in order to cast a ray 856 in the eyecamera coordinate system. In at least some embodiments, the eye cameracoordinate system 850 may have its origin at the 3D position of the eyetracking camera 324.

In FIG. 8C, 3D cornea center estimation module 716 simulates a cornealsphere 814 a (which may be based on assumed eye dimensions from database704) and corneal curvature center 816 a at a first position. The 3Dcornea center estimation module 716 may then check to see whether thecorneal sphere 814 a would properly reflect light from the light source326 to the glint position 854. As shown in FIG. 8C, the first positionis not a match as the ray 860 a does not intersect light source 326.

Similarly in FIG. 8D, 3D cornea center estimation module 716 simulates acorneal sphere 814 b and corneal curvature center 816 b at a secondposition. The 3D cornea center estimation module 716 then checks to seewhether the corneal sphere 814 b properly reflects light from the lightsource 326 to the glint position 854. As shown in FIG. 8D, the secondposition is also not a match.

As shown in FIG. 8E, the 3D cornea center estimation module 716eventually is able to determine the correct position of the cornealsphere is corneal sphere 814 c and corneal curvature center 816 c. The3D cornea center estimation module 716 confirms the illustrated positionis correct by checking that light from source 326 will properly reflectoff of the corneal sphere and be imaged by camera 324 at the correctlocation of glint 854 on image 852. With this arrangement and with theknown 3D positions of the light source 326, the camera 324, and theoptical properties of the camera (focal length, etc.), the 3D corneacenter estimation module 716 can determine the 3D location of thecornea's center of curvature 816 (relative to the wearable system).

The processes described herein in connection with at least FIGS. 8C-8Emay effectively be an iterative, repetitious, or optimization process toidentify the 3D position of the user's cornea center. As such, any of aplurality of techniques (e.g., iterative, optimization techniques, etc.)may be used to efficiently and quickly prune or reduce the search spaceof possible positions. Moreover, in some embodiments, the system mayinclude two, three, four, or more light sources such as light source 326and some of all of these light sources may be disposed at differentpositions, resulting in multiple glints such as glint 854 located atdifferent positions on image 852 and multiple rays such as ray 856having different origins and directions. Such embodiments may enhancethe accuracy of the 3D cornea center estimation module 716, as themodule 716 may seek to identify a cornea position that results in someor all of the glints & rays being properly reflected between theirrespective light sources and their respective positions on image 852. Inother words and in these embodiments, the positions of some or all ofthe light sources may be relied upon in the 3D cornea positiondetermination (e.g., iterative, optimization techniques, etc.) processesof FIGS. 8B-8E. In some implementations, the system may determine avector or ray along which the center of the cornea resides beforeperforming optimization processes (i.e., a 2D cornea center position).In such implementations, the 3D cornea center estimation module 716 mayonly search for cornea positions along such a vector, which may serve toprovide computational and/or time savings when performing optimizationprocesses. In at least some of these implementations, before determiningsuch a vector, the system may initially (i) define a first plane betweenthe origin of the eye camera coordinate system 850, a first light source(e.g., light source 326 a), and a first glint (e.g., glint 854 a)produced by the first light source, and (ii) define a second planebetween the origin of the eye camera coordinate system 850, a secondlight source (e.g., light source 326 b), and a second glint (e.g., glint854 b) produced by the second light source. The system may then simplycalculate the cross product of the first plane and the second plane todetermine the vector or ray along which the center of the cornea resides(i.e., the 2D cornea center position).

Example of Normalizing the Coordinate System of Eye Tracking Images

FIGS. 9A-9C illustrate an example normalization of the coordinate systemof eye tracking images, by a component in the wearable system such ascoordinate system normalization module 718 of FIG. 7A. Normalizing thecoordinate system of eye tracking images relative to a user's pupillocation may compensate for slippage of the wearable system relative toa user's face (e.g., headset slippage) and such normalization mayestablish a consistent orientation and distance between eye trackingimages and a user's eyes.

As shown in FIG. 9A, coordinate system normalization module 718 mayreceive estimated 3D coordinates 900 of a user's center of cornealrotation and may receive un-normalized eye tracking images such as image852. Eye tracking image 852 and coordinates 900 may be in anun-normalized coordinate system 850 that is based on the location of eyetracking camera 324, as an example.

As a first normalization step, coordinate system normalization module718 may rotate coordinate system 850 into rotated coordinate system 902,such that the z-axis (e.g., the vergence depth axis) of the coordinatesystem may be aligned with a vector between the origin of the coordinatesystem and cornea center of curvature coordinates 900, as shown in FIG.9B. In particular, coordinate system normalization module 718 may rotateeye tracking image 850 into rotated eye tracking image 904, until thecoordinates 900 of the user's corneal center of curvature are normal tothe plane of the rotated image 904.

As a second normalization step, coordinate system normalization module718 may translate rotated coordinate system 902 into normalizedcoordinate system 910, such that cornea center of curvature coordinates900 are a standard, normalized distance 906 from the origin ofnormalized coordinate system 910, as shown in FIG. 9C. In particular,coordinate system normalization module 718 may translate rotated eyetracking image 904 into normalized eye tracking image 912. In at leastsome embodiments, the standard, normalized distance 906 may beapproximately 30 millimeters. If desired, the second normalization stepmay be performed prior to the first normalization step.

Example of Locating a User's Pupil Centroid with an Eye Tracking System

FIGS. 9D-9G illustrate an example of locating a user's pupil center(e.g., the center of a user's pupil 822 as shown in FIG. 8A) using 3Dpupil center locator module 720 and eye tracking module 614.

As shown in FIG. 9D, 3D pupil center locator module 720 may receive anormalized eye tracking image 912 that includes a pupil centroid 913(e.g., a center of a user's pupil as identified by pupil identificationmodule 712). The 3D pupil center locator module 720 may then simulatethe normalized 3D position 910 of eye camera 324 to cast a ray 914 inthe normalized coordinate system 910, through the pupil centroid 913.

In FIG. 9E, 3D pupil center locator module 720 may simulate a cornealsphere such as corneal sphere 901 having center of curvature 900 basedon data from 3D cornea center estimation module 716 (and as discussed inmore detail in connection with FIGS. 8B-8E). As an example, the cornealsphere 901 may be positioned in the normalized coordinate system 910based on the location of the center of curvature 816 c identified inconnection with FIG. 8E and based on the normalization processes ofFIGS. 9A-9C. Additionally, 3D pupil center locator module 720 mayidentify a first intersection 916 between ray 914 (e.g., a ray betweenthe origin of normalized coordinate system 910 and the normalizedlocation of a user's pupil) and the simulated cornea, as shown in FIG.9E.

As shown in FIG. 9F, 3D pupil center locator module 720 may determinepupil sphere 918 based on corneal sphere 901. Pupil sphere 918 may sharea common center of curvature with corneal sphere 901, but have a smallerradius. 3D pupil center locator module 720 may determine a distancebetween cornea center 900 and pupil sphere 918 (e.g., a radius of pupilsphere 918) based on a distance between the corneal center and the pupilcenter. In some embodiments, the distance between a pupil center and acorneal center of curvature may be determined from assumed eyedimensions 704 of FIG. 7A, from eye tracking extrinsics and intrinsicsdatabase 702, and/or from per-user calibration data 706. In otherembodiments, the distance between a pupil center and a corneal center ofcurvature may be determined from per-user calibration data 706 of FIG.7A.

As shown in FIG. 9G, 3D pupil center locator module 720 may locate the3D coordinates of a user's pupil center based on variety of inputs. Asexamples, the 3D pupil center locator module 720 may utilize the 3Dcoordinates and radius of the pupil sphere 918, the 3D coordinates ofthe intersection 916 between a simulated cornea sphere 901 and a ray 914associated with a pupil centroid 913 in a normalized eye tracking image912, information on the index of refraction of a cornea, and otherrelevant information such as the index of refraction of air (which maybe stored in eye tracking extrinsics & intrinsics database 702) todetermine the 3D coordinates of the center of a user's pupil. Inparticular, the 3D pupil center locator module 720 may, in simulation,bend ray 916 into refracted ray 922 based on refraction differencebetween air (at a first index of refraction of approximately 1.00) andcorneal material (at a second index of refraction of approximately1.38). After taking into account refraction caused by the cornea, 3Dpupil center locator module 720 may determine the 3D coordinates of thefirst intersection 920 between refracted ray 922 and pupil sphere 918.3D pupil center locator module 720 may determine that a user's pupilcenter 920 is located at approximately the first intersection 920between refracted ray 922 and pupil sphere 918. With this arrangement,the 3D pupil center locator module 720 can determine the 3D location ofthe pupil center 920 (relative to the wearable system), in thenormalized coordinate system 910. If desired, the wearable system canun-normalize the coordinates of the pupil center 920 into the originaleye camera coordinate system 850. The pupil center 920 may be usedtogether with the corneal curvature center 900 to determine, among otherthings, a user's optical axis using optical axis determination module722 and a user's vergence depth by vergence depth estimation module 728.

Taking into account cornea refraction may possibly result in a morestable determined pupil position than one based on the firstintersection 916 between ray 914 (i.e., a ray between the origin ofnormalized coordinate system 910 and the normalized location of a user'spupil) and the simulated cornea, as shown in FIG. 9E. This is true inpart because, while simpler to compute, the first intersection 916 maynot correspond to a physical feature of the eye and, therefore, may notmove together with the eye as a solid body. In contrast, calculating thepupil center 920 by taking into account cornea refraction may correspondbetter to the physical pupil position of the eye, even if there is stillsome variation as a result of viewing angle. In various implements,therefore, determining the optical axis of the eye may thus involve acalculation of the true pupil center, not the first intersection 916between ray 914 and the simulated cornea.

A noticeable benefit in including refraction of the cornea occurs whenthe center of rotation (CoR) is estimated as a point at a fixed distancefrom the cornea center along the optical axis of the eye. In particular,including cornea refraction in determining pupil position maysignificantly reduce variation in calculating the center of rotation fordifferent orientations of the eye. For example, the variation can becaused when the eye as a whole moves in the camera coordinate frame,such as during remount of the headset because the eye as a whole may beoriented differently with respect to the headset on remount. Because thepupil center 920 corresponds better to the physical pupil position ofthe eye, there may be less variation in the CoR when the eye as a wholemoves in the camera coordinate frame. Advantageously, includingrefraction of the cornea surface may results in a more stable andaccurate CoR that can potentially be used in determining when a headsetis replaced onto a user's head, may allow for more correct render cameraplacement, may allow for other novel gaze tracking algorithms or anycombination thereof. Additionally, CoR as a stable, slowly changingfeature of the eye may potentially be tracked by multi-frame Kalman-typetemporal filters to provide a geometric reference location for otherapplications. FIG. 9H illustrates example calculated locations of apupil center that results from including cornea refraction and notincluding cornea refraction. When the pupil center is calculated withoutincluding the effect of cornea refraction, an external pupil 960 may bethe result. The external pupil center 960 may correspond to the firstintersection 916 between ray 914 (i.e., a ray between the origin ofnormalized coordinate system 910 and the normalized location of a user'spupil) and the simulated cornea, as shown in FIG. 9E. When the pupilcenter is calculated with taking cornea refraction into account, arefracted pupil center 962 may be the result. The refracted pupil center962 may correspond to the pupil center 920, as shown in FIG. 9G. Thedifferent locations of the external pupil center 960 and the refractedpupil center 962 may result in different calculations of the Center ofRotation of the eye, as determined as a fixed distance from the corneacenter 964 along the optical axis of the eye. For example, the externalpupil center 960 may result in an external rotation center 968. Theexternal rotation center 968 may vary significantly from the rotationcenter 966 calculated from the refracted pupil center 962.

FIGS. 9I-9L illustrate example experimental variations of the calculatedCenter of Rotation of the eye based on different calculated pupilcenters using a collection of over 400 datasets. The data selected wereonly those frames that had four glints and a valid pupil center (i.e.not all three x, y, z coordinate components being equal to zero), toexclude obvious pipeline failure cases. Different center of rotation(CoR) to cornea curvature center distance values (R) were examined, andit was found that R=4.7 mm gives a nearly optimal average result.However, specific user distance values can be adjusted to provide evensmaller variations of the CoR coordinates.

FIG. 9I illustrates a three dimensional graph of the x, y, and zcoordinates of the different pupil centers, and corresponding CoRscalculated using the different pupil centers for the dataset describedabove. FIG. 9J illustrates the data in FIG. 9I in the XY projection.Cluster 970 corresponds to the coordinates for the external pupil center960, cluster 972 corresponds to the coordinates for the correct,refracted pupil center 962, cluster 974 corresponds to the cornealocation (e.g., a three-dimensional cornea center location), cluster 978corresponds to the CoR using the correct, refracted pupil center 962,and cluster 980 corresponds to the CoR using the external pupil center960. Cluster 978 is smaller in size, specifically in the x directionthan cluster 980, indicating less variation in the CoR using therefracted pupil center 962 than in the CoR using the external pupilcenter 960.

TABLE 1 Left eye Right eye Center of mean mean mean mean mean mean meanmean Rotation sigma_(x) sigma_(y) sigma_(z) sigma_(3d) sigma_(x)sigma_(y) sigma_(z) sigma_(3d) 'external' 0.00056 0.00033 0.000410.00080 0.00058 0.00032 0.00037 0.00078 pupil 'refracted' 0.000250.00027 0.00044 0.00060 0.00027 0.00026 0.00040 0.00058 pupil

As can be seen from Table 1, the standard deviation (or sigma) ofx-component of the CoR is reduced by about half when the refracted pupilcenter 962 was used for the calculation as compared to when the externalpupil center 960. The total three dimensional standard deviation(sigma_(3d)) also reduced significantly with the use of the refractedpupil center 962.

FIGS. 9K and 9L illustrates the mean and median CoR standard deviation,respectively, as a function of the CoR to cornea center of curvaturedistance for a collection of over 400 datasets. Graphs 991L and 991Rillustrate the mean three dimensional CoR standard deviation(sigma_(3d)) as a function of the CoR to cornea center of curvaturedistance for the left and right eye, respectively. Graphs 992L and 992Rillustrate the median three dimensional CoR standard deviation(sigma_(3d)) of the CoR to cornea center of curvature distance for theleft and right eye respectively. Curve 982A corresponds to the totalleft eye mean sigma_(3d), curve 982B corresponds to the total right eyemean sigma_(3d), curve 982C corresponds to the total left eye mediansigma_(3d), curve 982D corresponds to the total right eye mediansigma_(3d). Curves 984A-D correspond to the x components of the varioussigma_(3d), curves 990A-D correspond to the y components of the varioussigma_(3d), and curves 986A-D correspond to the z components of thevarious sigma_(3d).

FIG. 9M illustrates an example distribution of the optimal radiuscalculated individually for each user dataset as a value of the CoR tocornea distance that provides a minimum of the sigma_(3d). The mean ofthe distribution is found to be at R=4.9 mm, with a standard deviationof 1.5 mm. The users with very small radius (R˜1 mm) were found to becases with very bad gaze tracking and so had to be excluded.

Example of Differences Between Optical and Visual Axes

As discussed in connection with optical to visual mapping module 730 ofFIG. 7A, a user's optical and visual axes are generally not aligned, duein part to a user's visual axis being defined by their fovea and thatfoveae are not generally in the center of a person's retina. Thus, whena person desires to concentrate on a particular object, the personaligns their visual axis with that object to ensure that light from theobject falls on their fovea while their optical axis (defined by thecenter of their pupil and center of curvature of their cornea) isactually slightly offset from that object. FIG. 10 is an example of aneye 1000 illustrating the eye's optical axis 1002, the eye's visual axis1004, and the offset between these axes. Additionally, FIG. 10illustrates the eye's pupil center 1006, the eye's center of corneacurvature 1008, and the eye's average center of rotation (CoR) 1010. Inat least some populations, the eye's center of cornea curvature 1008 maylie approximately 4.7 mm in front, as indicated by dimension 1012, ofthe eye's average center of rotation (CoR) 1010. Additionally, the eye'scenter of perspective 1014 may lie approximately 5.01 mm in front of theeye's center of cornea curvature 1008, about 2.97 mm behind the outersurface 1016 of the user's cornea, and/or just in front of the user'spupil center 1006 (e.g., corresponding to a location within the anteriorchamber of eye 1000). As additional examples, dimension 1012 may between3.0 mm and 7.0 mm, between 4.0 and 6.0 mm, between 4.5 and 5.0 mm, orbetween 4.6 and 4.8 mm or any ranges between any values and any valuesin any of these ranges. The eye's center of perspective (CoP) 1014 maybe a useful location for the wearable system as, in at least someembodiments, registering a render camera at the CoP may help to reduceor eliminate parallax artifacts.

FIG. 10 also illustrates such a within a human eye 1000 with which thepinhole of a render camera can be aligned. As shown in FIG. 10 , thepinhole of a render camera may be registered with a location 1014 alongthe optical axis 1002 or visual axis 1004 of the human eye 1000 closerto the outer surface of the cornea than both (a) the center of the pupilor iris 1006 and (b) the center of cornea curvature 1008 of the humaneye 1000. For example, as shown in FIG. 10 , the pinhole of a rendercamera may be registered with a location 1014 along the optical axis1002 of the human eye 1000 that is about 2.97 millimeters rearward fromthe outer surface of the cornea 1016 and about 5.01 millimeters forwardfrom the center of cornea curvature 1008. The location 1014 of thepinhole of the render camera and/or the anatomical region of the humaneye 1000 to which the location 1014 corresponds can be seen asrepresenting the center of perspective of the human eye 1000. Theoptical axis 1002 of the human eye 1000 as shown in FIG. 10 representsthe most direct line through the center of cornea curvature 1008 and thecenter of the pupil or iris 1006. The visual axis 1004 of the human eye1000 differs from the optical axis 1002, as it represents a lineextending from the fovea of the human eye 1000 to the center of thepupil or iris 1006.

Example Processes of Rendering Content and Checking Registration Basedon Eye Tracking

FIG. 11 is a process flow diagram of an example method 1100 for usingeye tracking in rendering content and providing feedback on registrationin a wearable device. The method 1100 may be performed by the wearablesystem described herein. Embodiments of the method 1100 can be used bythe wearable system to render content and provide feedback onregistration (e.g., fit of the wearable device to the user) based ondata from an eye tracking system.

At block 1110, the wearable system may capture images of a user's eye oreyes. The wearable system may capture eye images using one or more eyecameras 324, as shown at least in the example of FIG. 3 . If desired,the wearable system may also include one or more light sources 326configured to shine IR light on a user's eyes and produce correspondingglints in the eye images captured by eye cameras 324. As discussedherein, the glints may be used by an eye tracking module 614 to derivevarious pieces of information about a user's eye including where the eyeis looking.

At block 1120, the wearable system may detect glints and pupils in theeye images captured in block 1110. As an example, block 1120 may includeprocessing the eye images by glint detection & labeling module 714 toidentify the two-dimensional positions of glints in the eye images andprocessing the eye images by pupil identification module 712 to identifythe two-dimensional positions of pupils in the eye images.

At block 1130, the wearable system may estimate the three-dimensionalpositions of a user's left and right corneas relative to the wearablesystem. As an example, the wearable system may estimate the positions ofthe center of curvature of a user's left and right corneas as well asthe distances between those centers of curvature and the user's left andright corneas. Block 1130 may involve 3D cornea center estimation module716 identifying the position of the centers of curvature as describedherein at least in connection with FIGS. 7A and 8A-8E.

At block 1140, the wearable system may estimate the three-dimensionalpositions of a user's left and right pupil centers relative to thewearable system. As an example, the wearable system and 3D pupil centerlocator module 720 in particular, may estimate the positions of theuser's left and right pupil centers as described at least in connectionwith FIGS. 7A and 9D-9G, as part of block 1140.

At block 1150, the wearable system may estimate the three-dimensionalpositions of a user's left and right centers or rotation (CoR) relativeto the wearable system. As an example, the wearable system and CoRestimation module 724 in particular, may estimate the positions of theCoR for the user's left and right eyes as described at least inconnection with FIGS. 7A and 10 . As a particular example, the wearablesystem may find the CoR of an eye by walking back along the optical axisfrom the center of curvature of a cornea towards the retina.

At block 1160, the wearable system may estimate a user's IPD, vergencedepth, center of perspective (CoP), optical axis, visual axis, and otherdesired attributes from eye tracking data. As examples, IPD estimationmodule 726 may estimate a user's IPD by comparing the 3D positions ofthe left and right CoRs, vergence depth estimation module 728 mayestimate a user's depth by finding an intersection (or nearintersection) of the left and right optical axes or an intersection ofthe left and right visual axes, optical axis determination module 722may identify the left and right optical axes over time, optical tovisual axis mapping module 730 may identify the left and right visualaxes over time, and the CoP estimation module 732 may identify the leftand right centers of perspective, as part of block 1160.

At block 1170, the wearable system may render content and may,optionally, provide feedback on registration (e.g., fit of the wearablesystem to the user's head) based in part on the eye tracking dataidentified in blocks 1120-1160. As an example, the wearable system mayidentify a suitable location for a render camera and then generatecontent for a user based on the render camera's location, as discussedin connection with light-field render controller 618, FIG. 7B, andrender engine 622. As another example, the wearable system may determineif it is properly fitted to the user, or has slipped from its properlocation relative to the user, and may provide optional feedback to theuser indicating whether the fit of the device needs adjustment, asdiscussed in connection with registration observer 620. In someembodiments, the wearable system may adjust rendered content based onimproper or less than ideal registration in an attempt to reduce,minimize or compensate for the effects of improper or mis-registration.

Example Graphs of Rendering Content in Response to User Eye Movements

FIG. 12 includes a set of example graphs 1200 a-1200 j which illustratehow the wearable system may switch depth planes in response to theuser's eye movements. As discussed herein in connection with FIGS. 4 and7 , the wearable system may include multiple depth planes, where thevarious depth planes are configured to present content to a user at adifferent simulated depth or with a different accommodation cue (e.g.,with various levels of wavefront curvature or light ray divergence). Asan example, the wearable system may include a first depth planeconfigured to simulate a first range of depths and a second depth planeconfigured to simulate a second range of depths and, while these tworanges may desirably overlap to facilitate hysteresis in switching, thesecond range of depths may generally extend to greater distances fromthe user. In such embodiments, the wearable system may track a user'svergence depth, saccade movements, and blinks to switch between thefirst and second depth planes in a manner that avoid excessive depthplane switching, excessive accommodation-vergence mismatches, andexcessive periods of accommodation-vergence mismatch and that seek toreduce the visibility of depth plane switches (e.g., by shifting depthplanes during blinks and saccades).

Graph 1200 a illustrates an example of a user's vergence depth overtime. Graph 1200 b illustrates an example of a user's saccade signal orvelocity of eye movements over time.

Graph 1200 c may illustrate vergence depth data generated by eyetracking module 614 and, in particular, data generated by vergence depthestimation module 728. As shown in graphs 1200 c-1200 h, eye trackingdata may be sampled within eye tracking module 614 at a rate ofapproximately 60 Hz. As shown between graphs 1200 b and 1200 c, eyetracking data within eye tracking module 614 may lag behind a user'sactual eye movements by a delay 1202. As an example, at time t₁ a user'svergence depth may cross a hysteresis threshold 1210 a, but thehysteresis band crossing detection module 752 may not recognize theevent until time t₂ after delay 1202.

Graph 1200 c also illustrates various thresholds 1210 a, 1210 b, 1210 cin a hysteresis band, which may be associated with transitions betweenfirst and second depth planes (e.g., depth planes #1 and #0 in FIG. 12). In some embodiments, the wearable system may try to display contentwith depth plane #1 whenever a user's vergence depth is greater thanthreshold 1210 b and to display content with depth plane #0 whenever auser's vergence depth is less than threshold 1210 b. However, to avoidexcessive switching, the wearable system may implement hysteresis,whereby the wearable system will not switch from depth plane #1 to depthplane #0 until the user's vergence depth crosses outer threshold 1210 c.Similarly, the wearable system may not switch from depth plane #0 todepth plane #1 until the user's vergence depth crosses outer threshold1210 a.

Graph 1200 d illustrates an internal flag that may be generated by depthplane selection module 750, or hysteresis band crossing detection module752, indicating whether the user's vergence depth is in the volumegenerally associated with depth plane #1 or the volume generallyassociated with depth plane #2 (e.g., whether the user's vergence depthis greater or less than threshold 1210 b).

Graph 1200 e illustrates an internal hysteresis band flag that may begenerated by depth plane section module 750, or hysteresis band crossingdetection module 752, indicating whether a user's vergence depth hascross an outer threshold such as threshold 1210 a or 1210 c. Inparticular, graph 1200 e illustrates a flag indicative of whether theuser's vergence depth has completely crossed a hysteresis band and intoa region outside of the active depth plane's volume (e.g., into a regionassociated with a depth plane other than an active depth plane), thuspotentially leading to undesirable accommodation-vergence mismatch(AVM).

Graph 1200 f illustrates an internal AVM flag that may be generated bydepth plane selection module 750, or hysteresis band crossing detectionmodule 752, indicating whether a user's vergence has been in outside ofthe active depth plane's volume for greater than a predetermined time.The AVM flag may therefore identify when the user may have beensubjected to an undesirable accommodation-vergence mismatch for anearly-excessive or excessive period of time. Additionally oralternatively, the internal AVM flag may also indicate whether a user'svergence has gone a predetermined distance beyond the active depthplane's volume, thus creating a potentially-excessiveaccommodation-vergence mismatches. In other words, the AVM flag mayindicate when a user's vergence has exceeded an additional thresholdeven further from threshold 1210 b than thresholds 1210 a and 1210 c.

Graph 1200 g illustrates an internal blink flag that may be generated byocular event detection module 754, which may determine when a user hasor is blinking. As noted herein, it may be desired to switch depthplanes upon user blink, to reduce the likelihood of the user perceivingthe switch in depth planes.

Graph 1200 h illustrates an example output from depth plane selectionmodule 750. In particular, graph 1200 h shows that depth plane selectionmodule 750 may output an instruction to utilize a selected depth plane,which may change over time, to a render engine such as render engine 622(see FIG. 6 ).

Graphs 1200 i and 1200 j illustrate delays that may be present in thewearable system including a delay by render engine 622 to switch depthplanes and a delay by the display 220, which may need to provide lightassociated with a new image frame in a new depth plane to effectuate achange in depth planes.

Reference will now be made to the events illustrated in graphs 1200a-1200 j at various times (t₀-t₁₀).

Sometime around time t₀, a user's vergence depth may cross threshold1210 a, which may be an outer hysteresis threshold. After a delayassociated with image capture and signal processing, the wearable systemmay generate a signal, as indicated in graph 1200 e, that indicates thatthe user's vergence depth lies within the hysteresis band. In theexample of graph 1200 e, one or more modules of light-field rendercontroller 618 may present a hysteresis band exceeded flag atapproximately time t₁ in connection with the user's vergence depthcrossing threshold 1210 a.

The user's vergence depth may continue to decrease from time to untilapproximately time t₄ and may thereafter increase.

At time t₁, a user's vergence depth may cross threshold 1210 b, whichmay be a midpoint between two depth planes such as depth planes #1 and#0. After processing delay 1202, eye tracking module 614 may alter aninternal flag indicating that the user's vergence depth has moved from avolume generally associated with depth plane #1 into a volume generallyassociated with depth plane #0, as illustrated in graph 1200 d.

At time t₃, one or more modules of light-field render controller 618 maydetermine that the user's vergence depth, as shown in graph 1200 a, hasmoved entirely through the hysteresis band and cross outer threshold1210 c. As a result, one or more modules of light-field rendercontroller 618 may generate a signal, as indicated in graph 1200 e, thatindicates that the user's vergence depth lies outside the hysteresisband. In at least some embodiments, one or more modules of light-fieldrender controller 618 (e.g., depth plane selection module 750) mayswitch between first and second depth planes only when a user's vergencedepth is outside of the hysteresis band between those two depth planes.

In at least some embodiments, one or more modules of light-field rendercontroller 618 may be configured to switch depth planes at time t₃. Inparticular, one or more modules of light-field render controller 618 maybe configured to switch depth planes based on a determination that thevergence depth has moved from the volume of the currently selected depthplane (depth plane #1 as indicated by graph 1200 h) into the volume ofanother depth plane (depth plane #0) and entirely crossed a hysteresisband. In other words, one or more modules of light-field rendercontroller 618 may implement a depth plane switch whenever thehysteresis band is exceeded (graph 1200 e is high) and anaccommodation-vergence mismatch based on time or magnitude of mismatchis detected (graph 1200 f is high). In such embodiments, one or moremodules of light-field render controller 618 may provide a signal torender engine 622 instructing render engine 622 to switch to the otherdepth plane (depth plane #0). In the example of FIG. 12 , however, oneor more modules of light-field render controller 618 may be configuredto delay depth plane switches until at least one other condition hasbeen satisfied. These additional conditions may include, as examples, ablink condition, an accommodation-vergence mismatch timeout condition,and an accommodation-vergence magnitude condition.

At time t₄ and in the example of FIG. 12 , one or more modules oflight-field render controller 618 may be configured to switch depthplanes. In particular, one or more modules of light-field rendercontroller 618 may determine that the user's vergence has been in thevolume associated with depth plane #0 for longer than a predeterminedthreshold of time (and optionally, also outside of the hysteresis bandfor that period of time). Examples of predetermined thresholds of timeinclude 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, and 90seconds and any range between any of these values. Upon such adetermination, one or more modules of light-field render controller 618may generate an AVM flag, as indicated in graph 1200 f, and directrender engine 622 to switch to depth plane #0, as indicated in graph1200 h. In some embodiments, one or more modules of light-field rendercontroller 618 may generate an AVM flag and direct render engine 622 toswitch depth planes if the user's vergence depth is detected to be morethan a threshold distance from the current selected depth volume.

At time t₅ and after delay 1204, the render engine 622 may startrendering content at the newly-selected depth plane #0. After a delay1206 associated with rendering and conveying light to a user through thedisplay 220, the display 220 may be fully switched to the newly-selecteddepth plane #0 by time t₆.

Thus, graphs 1200 a-j illustrates, between times to and t₆, how thesystem may respond to a user's changing vergence and may switch depthplanes after the user's vergence has moved away from a prior depthvolume for more than a predetermined period of time. Graphs 1200 a-j,between times t₇ and t₁₀, may illustrate how the system responds to auser's changing vergence and may switch depth planes upon detection ofthe user blinking, which may be prior to the predetermined period oftime.

At time t₇, one or more modules of light-field render controller 618 maydetect that the user's vergence depth has entered the hysteresis regionbetween depth planes #0 and #1 (e.g., that the user's vergence depth hascrossed outer threshold 1210 c). In response, one or more modules oflight-field render controller 618 may alter a hysteresis flag as shownin graph 1200 e.

At time t₈, one or more modules of light-field render controller 618 maydetect that the user's vergence depth has cross threshold 1210 b andmoved from the volume generally associated with depth plane #0 into thevolume generally associated with depth plane #1. As such, one or moremodules of light-field render controller 618 may alter a depth volumeflag, as shown in graph 1200 d.

At time t₉, one or more modules of light-field render controller 618 maydetect that the user's vergence depth has crossed threshold 1210 a andmoved out of the hysteresis volume into the volume generally associatedexclusively with depth plane #1. In response, one or more modules oflight-field render controller 618 may alter a hysteresis flag as shownin graph 1200 e.

At around time t₁₀, the user may blink and one or more modules oflight-field render controller 618 may detect that blink. As one example,ocular event detection module 754 may detect a user's blink. Inresponse, one or more modules of light-field render controller 618 maygenerate a blink flag, as shown in graph 1200 h. In at least someembodiments, one or more modules of light-field render controller 618may implement a depth plane switch whenever the hysteresis band isexceeded (graph 1200 e is high) and a blink is detected (graph 1200 g ishigh). Thus, one or more modules of light-field render controller 618may instruct render engine 622 to switch depth planes at time t₁₀.

Example Rendering Modes in a Mixed Reality System Having Multiple DepthPlanes

In mixed reality systems, computer-generated (rendered) scenes may beconveyed to the human eye such that real and virtual objects arespatially aligned (from the perspective of the user). To provide a userwith a visual perception of spatial alignment between real and virtualobjects, the perspective from which the computer-generated scene isrendered and presented may preferably correspond to the perspective(e.g., the position and orientation) of the user's eye. As an example,user may perceive real and virtual objects to be spatially aligned in adesired manner when a “Real World” frame (within which real objectsexist) and a “Render World” frame (within which virtual objects exist)are accurately aligned with one another.

A digital light-field display device, such as wearable system 200including display 220 of FIG. 2 , is an example of a mixed realitysystem in which a light-field representative of 3D virtual content(virtual objects) may be provided to a user at various depths using oneor more depth planes. The depth planes may be compared to one or morevirtual screens, at varying distances from the user, onto which virtualcontent can be projected or displayed, converted into virtual pixels,and provided to a user. As such, a mixed reality system may be opticallyequivalent to a system having one or more transparent floating screens,located at varying distances from the user. In this manner, a digitizedlight field is projected through a user's iris onto their retina and animage of the 3D virtual content is formed (e.g., the user perceives theimage of the 3D virtual content).

FIG. 13 shows a mixed reality system 1300 in which a light-fieldrepresentative of 3D virtual content including virtual objects isprovided to a user's eye using one or more depth planes, which may beoptical structures on a wearable device that simulate virtual screensspaced apart from the user and the wearable device at various distances.The mixed reality system 1300 may include an eyepiece 1310, which mayrepresent an eyepiece of a head-worn digital light-field display device,such as display 220 of FIG. 3 , or a portion thereof. Such a system maybe configured to project a light-field representative of 3D virtualobject 1330 through the eyepiece 1310 and onto the retina 1303 of auser's eye 1302.

FIG. 13 also shows depth planes 1321-1323 onto which the virtual object1330 and other virtual content can be projected or displayed andconverted into virtual pixels. In the particular example depicted inFIG. 13 , the virtual object 1330 is projected onto depth plane 1322 andthereby converted into virtual pixels 1332. As a result, light generatedby eyepiece 1310 (e.g., display 220) may provide accommodation cues tothe user's eye 1302 as if the virtual object 1330 were provided on aphysical display or projector screen located at the distance from theuser of depth plane 1322. The head-worn digital display device maygenerate a digitized light field representative of the virtual pixels1332, and may project such a light field through the eyepiece 1310 andonto the retina 1303 of a user's eye 1302.

As will be discussed in more detail below, different rendering modes maybe employed in a mixed reality system (such as mixed reality system 1300in FIG. 13 ) to provide differing eye accommodation cues across a user'sfield of view, for different content, and/or at different periods oftime. As examples, a mixed reality system may employ a discretevari-focus mode in which virtual objects are displayed using a singledepth plane at a time (as shown in FIGS. 14A-14B), may employ a blendedvari-focus mode in which virtual objects are displayed using twoadjacent depth planes to generate accommodation cues between the twodepth planes (as shown in FIGS. 14C-14D), and may employ a multi-focusmode in which virtual objects are displayed using two or more depthsplanes to generate two or more accommodation cues simultaneously (asshown in FIGS. 14E-14F). In general, a mixed reality system may switchbetween these and other rendering modes during operation in response toa variety of conditions. As an example, the mixed reality system mayutilize a first rendering mode (such as the discrete vari-focus mode)when displaying a first kind of content (such as text that may beprovided at a single depth) and may utilize a second, differentrendering mode (such as the multi-focus mode) when displaying a secondkind of content (such as content that may be provided at a variety ofdepths simultaneously). In at least some embodiments, light-field rendercontroller 618 of FIG. 6 may be configured to select which renderingmode is employed at any given time, based on a variety of inputs andconditions as discussed herein.

Single Depth Plane Rendering Mode (Discrete Vari-Focus Mode)

As illustrated in FIGS. 14A and 14B, the wearable system describedherein can render virtual reality objects using a single depth plane ata time, in mode referred to herein as a single depth plane renderingmode and also referred to as a discrete-vari-focus mode. In the discretevari-focus mode, the wearable system may utilize a single depth plane,across the entire field-of-view (FOV) of the display, for displaying allof the currently-rendered virtual objects (even if some of those objectsare disposed at depths other than those associated with that depthplane). In other words, the wearable system may provide a single focalor accommodation cue across the entire FOV. Of course, the wearablesystem may switch which depth plane is used to render content over time,thus altering the accommodation cue of rendered virtual content overtime (in response to changes in user vergence depth, the depth ofvirtual content, and other facts as discussed in more detail herein). Ingeneral, in the discrete vari-focus mode, only one depth plane isemployed by the mixed reality system at any one time.

FIGS. 14A-14B respectively show top and side/isometric views of a mixedreality system 1400A that is operating in discrete vari-focus mode topresent content to a user's eye 1402. FIGS. 14A-14B also shows depthplanes 1421, 1422, and 1423 onto which virtual content can be projectedand converted into virtual pixels. While operating in discretevari-focus mode, virtual content may only be projected onto one of depthplanes 1421-1423 at a given time. In the example of FIGS. 14A-14B, themixed reality system 1400A has been switched to a state in which virtualcontent is projected onto depth plane 1422, but is not projected ontoeither of depth planes 1421 or 1423. As shown, the virtual content thatis projected onto depth plane 1422 is converted into virtual pixels1432A.

As discussed in more detail in connection with FIGS. 4, 6, 7A, 7B, and12 , the choice of which depth to place all the pixels on (e.g., whichdepth plane virtual content is to be projected onto) and the timing ofswitching between depth planes may be based on eye tracking and virtualcontent information. As examples, the wearable system, while operatingin a discrete vari-focus mode, may switch which depth plane is activebased on a user's vergence depth, based a user's vergence depth but withdepth plane switches delayed until a trigger event (e.g., upon userblink or saccade, after a predetermined AVM mismatch timeout asdiscussed in connection with FIG. 12 , etc.), based on the depth ofvirtual content, and based on the depth of virtual content but withdepth plane switches delayed until a trigger event (e.g., upon userblink or saccade, after a predetermined AVM mismatch timeout asdiscussed in connection with FIG. 12 , etc.) or any combination ofthese. In at least some embodiments, one or more of depth planeselection module 750, hysteresis band crossing detection module 752, andocular event detection module 754, individually or in combination, maybe configured to implement a desired depth plane switching scheme.

Blended Depth Plane Rendering Mode (Blended Vari-Focus Mode)

As illustrated in FIGS. 14C and 14D, the wearable system describedherein can render virtual reality objects using two adjacent depthplanes to generate accommodation or focal cues that lie between depthplanes. In some embodiments, this blended vari-focus mode may enable thewearable system to generate accommodation or focal cues at any distancebetween, and including, the depths provided by the set of depth planesin the wearable system. In other words, if the system includes threedepth planes, a first of which provides an accommodation cue of 1 foot,a second of which provides an accommodation cue of 10 feet, and a thirdof which provides an accommodation cue of optical infinity, then thewearable system may be able to provide the user with accommodation cuesanywhere from the 1 foot depth of the first plane to the opticalinfinity depth of the third depth plane.

FIGS. 14C-14D show top and side/isometric views of a mixed realitysystem 1400C that is operating in blended vari-focus mode to presentcontent to the user's eye 1402. The mixed reality system 1400C may, forexample, have the same architecture as the mixed reality system 1400A asdescribed above with reference to FIGS. 14A-14B. While operating inblended vari-focus mode, virtual content can be simultaneously projectedonto two or more of depth planes 1421-1423 at any given point in time soas to generate an accommodation cue between planes. In the example ofFIGS. 14C-14D, the mixed reality system 1400C has been switched to astate in which virtual content is projected onto depth planes 1421 and1422. As shown, the virtual content that is projected onto depth planes1421 and 1422 is converted into virtual pixel sets 1431C and 1432C,respectively. Virtual pixel sets 1431C and 1432C may blend together and,based on their relative intensities, provide the user with anaccommodation cue somewhere between depth planes 1421 and 1422.

In the blended vari-focus mode, the wearable system may provide the samefocal or accommodation cue for all pixels across the display's FOV, andthis accommodation cue may be continuously variable between the depthsof any pair of adjacent depth planes. The wearable system may achievecontinuously variable accommodation cues by blending pixel intensitiesbetween two depth planes. As an example, the wearable system may displaya virtual object having an accommodation cue between depth planes 1422and 1421 by rendering the virtual object in both depth planes 1421 and1421. In the further example in which the virtual object is closer tothe depth of depth plane 1421, the wearable system may render thevirtual object at a greater light intensity (e.g., brightness) in depthplane 1421 than in depth plane 1422. In such an arrangement, the lightfrom the two depth planes may blend such that the user perceives thevirtual object as having an accommodation cue that lies near depth plane1421 (but still between planes 1421 and 1422).

In the blended vari-focus mode, the wearable system is configured toselect which adjacent depth planes to blend, to provide a desiredaccommodation cue. However, since the accommodation cue can varycontinuously between the planes by continuously varying the brightness,the timing of depth plane switches may not be as significant as in thediscrete vari-focus mode. Thus, the wearable system may be configured toswitch which two depth planes form the pair of adjacent depth planeswithout waiting for a triggering event such as a user blink, saccade, orAVM timeout. Instead, the wearable system may smoothly vary the providedaccommodation cue, and which depth planes are utilized, over time inresponse to a user's vergence depth, to the depth of virtual content, orto a combination of these and other inputs as desired.

Multiple Depth Plane Rendering Mode (Multi-Focus Mode)

As illustrated in FIGS. 14E and 14F, the wearable system describedherein can render virtual reality objects, in a multi-focus mode, usingtwo or more depth planes to generate two or more accommodation cuessimultaneously. In other words, virtual content in a given frame may bepresented across multiple depths simultaneously. As examples, themulti-focus mode may include using two or more depth planes, in themanner described in connection with the blended vari-focus mode, toprovide a first blended accommodation cue; using two or more depthplanes to provide a second blended accommodation cue; using a singledepth plane to provide a third accommodation cue; using a second singledepth plane to provide a fourth accommodation cue; or using combinationsof these and other focus modes to provide a variety of accommodationcues.

FIGS. 14E-14F show top and side/isometric views of a mixed realitysystem 1400E that is operating in multi-focus mode to present content tothe user's eye 1402. The mixed reality system 1400E may, for example,have the same architecture as the mixed reality system 1400A and/or1400C as described above with reference to FIGS. 14A-14D. Whileoperating in multi-focus mode, virtual content can be simultaneouslyprojected onto two or more of depth planes 1421-1423 at any given pointin time so as to generate two or more different focus cues. In theexample of FIGS. 14E-14F, virtual content is simultaneously projectedonto depth planes 1421, 1422, and 1423. As shown, the virtual contentthat is projected onto depth planes 1421-1423 is converted into virtualpixel sets 1431E-1433E,

In general, the wearable system can provide by blended and non-blendedaccommodation cues while operating in a multi-focus mode (or whileoperating in a blended vari-focus mode). As shown in FIG. 14F, depthplanes 1421 and 1422 may be configured to provide content 1450 with ablended accommodation cue using pixels 1431E on depth plane 1421 andpixels 1432E on depth plane 1422, may be configured to provide content1452 with a different blended accommodation cue using pixels 1432F ondepth plane 1422 and pixels 1433F on depth plane 1423, and may beconfigured to provide content 1454 with a non-blended accommodation cueusing pixels 1433G on depth plane 1423. Blended content 1452 may includecontent that stretches across the depth dimension of the user'sperspective and thus includes portions at a depth associated with depthplane 1423 rendered as virtual pixels 1433F and additional portions at adifferent depth associated with depth plane 1422 rendered as virtualpixels 1432F. As an example, blended content 1542 may includenon-overlapping content, such that virtual pixels 1432F do not overlap,in the planes perpendicular to the depth axis, with the virtual pixels1433F. In other embodiments, blended content 1452 may include content atdepths between planes 1422 and 1423, which is rendered by overlappingand blending virtual pixels 1433F and 1432F. Overlapping and blendingpixels 1433F and 1432F may include varying the relative intensities ofpixels 1433F and 1432F to shift the apparent depth towards depth plane1422 or towards depth plane 1422. As an example, dimming pixels 1433Fand amplifying or brightening pixels 1432F may have the effect ofshifting the user perceived depth towards depth plane 1422. Moreover,overlapped pixels may have a different intensity (which may generally bea lower intensity) than non-overlapped pixels, such that overlapped andblended pixels have a desired intensity (to prevent overlapped pixelsfrom appearing excessively bright due to the contribution of multipledepth planes to light at that position in the display). These are merelyillustrative examples and, in general, the wearable system may presentany desired combination of blended and non-blended accommodation cues.

As discussed in connection with the blended vari-focus mode, the timingof depth plane switches may not be as significant in modes with variableaccommodation cues, such as the blended vari-focus mode and themulti-focus mode, as such switches are in a discrete vari-focus mode.Thus, the wearable system may be configured to switch which depth planesare active in the multi-focus mode without waiting for a triggeringevent such as a user blink, saccade, or AVM timeout. Instead, thewearable system may smoothly vary the provided accommodation cues, andwhich depth planes are utilized, over time in response to a user'svergence depth, to the depth of virtual content, or to a combination ofthese and other inputs as desired. In other implementations, however,triggering events such as a user blink, saccade, or AVM timeout may beutilized.

Effects of Center of Perspective Misalignments in Various RenderingModes

When projected onto one or more depth planes, it may be desirable torender and view 3D virtual content from a particular center ofperspective (CoP), which may be determined for both a render world andthe real world. When content is rendered from a proper center ofperspective in the render world, the pixels of each virtual screen mayaccurately appear as 3D virtual content when observed from a propercenter of perspective in the real word, which may include a specificposition and orientation. However, if the same content is rendered froma different position in the render world or viewed from a differentposition in the real world, the 3D virtual content may not accuratelyresemble an image of such 3D virtual content. This rendering frameworkcan be represented using a pinhole camera model, where the CoP isrepresented as a “virtual” or “render” pinhole camera that is positionedand oriented within the Render World (e.g., 3D render space) in a mannerso as to correctly capture the projection of the 3D virtual content.Additional detailed related to the CoP is described below, with respectto FIGS. 22-24B, while FIGS. 15A-15C and corresponding descriptionprovided below further demonstrate the impact that CoP accuracy and/orprecision may have on mixed reality system performance for each ofvarious example rendering modes.

In operation, a digitized light field that is projected onto the retinaof the user's eye may serve to form an image of 3D virtual contentcontaining artifacts or exhibiting other problematic characteristics ifthe perspective of the virtual pinhole camera (in the Render Worldframe) and the perspective of the user (in the Real World frame) aremisaligned. For a simple scenario in which the one depth plane isemployed, a misalignment between the virtual pinhole camera and theperspective of the user may yield a digitized light field that forms animage of 3D virtual content at an incorrect (an unintended) locationwithin the user's FOV. For scenarios in which two or more depth planesare employed, such as scenarios in which any of the focus modesdescribed above with reference to FIGS. 14A-14F are employed, amisalignment between the virtual pinhole camera and the perspective ofthe user may yield a digitized light field that forms an image of 3Dvirtual content that appears to jump or pop between locations and/orthat includes visual artifacts, such as fractures and/or dislocations.More specifically, an occurrence of such a misalignment in a mixedreality system operating in any one of the above-mentioned focus modesmay serve to introduce a parallax shift that causes 3D virtual contenthaving been planarized onto one depth plane (or blended between depthplanes) to shift out of place relative to 3D virtual content having beenplanarized onto another depth plane (or blended between depth planes).

As an example of CoP misalignment, FIG. 15A shows a mixed reality system1500A that is operating in a discrete vari-focus mode to present contentto a user's eye 1502 in the presence of a CoP misalignment. Much likemixed reality system 1300, as described above with reference to FIG. 13, the mixed reality system 1500A includes an eyepiece 1510, which mayrepresent that of a head-worn digital light-field display device or aportion thereof that is configured to project a light-fieldrepresentative of 15D virtual object 1530 through the eyepiece 1510 andonto the retina 1503 of a user's eye 1502. FIG. 15A also shows depthplanes 1521-1523 onto which the virtual object 1530 can be projected andconverted into virtual pixels. In the particular example depicted inFIG. 15A, the mixed reality system 1500A operates in a discretevari-focus mode (e.g., in a manner similar to that having been describedabove with reference to FIGS. 14A-14B) to switch from projecting virtualobject 1530 onto depth plane 1522 at a first point in time to projectingvirtual object 1530 onto depth plane 1523 at a second, subsequent pointin time. As such, the virtual object 1530 is projected onto depth plane1522 and converted into virtual pixels 1532A₁ at the first point intime, and is then projected onto depth plane 1523 and converted intovirtual pixels 1533A₂ at the second point in time. It follows that themixed reality system 1500A may generate and project a digitized lightfield representative of the virtual pixels 1532A₁ through the eyepiece1510 and onto the retina 1503 of a user's eye 1502 at the first point intime, and may generate and project a digitized light fieldrepresentative of the virtual pixels 1533A₂ through the eyepiece 1510and onto the retina 1503 of a user's eye 1502 at the second point intime.

Although there may be no desire to perceptually change the image of thevirtual object 1530 formed on the retina 1503 of the user's eye 1502between the first and second points in time (other than providing a newaccommodation cue), because there is a CoP misalignment in the exampleof FIG. 15A (e.g., misalignment between the perspective of the virtualcamera and the correct perspective of the user's eye 1502), a parallaxshift may occur as the mixed reality system 1500A switches depth planesthat may serve to change the perceived location of the virtual 1530within the user's FOV. FIG. 15A further shows example retinal images1534A₁ and 1534A₂ that are representative of images of the virtualobject 1530 formed on the retina 1503 of the user's eye 1502 at thefirst and second points in time, respectively. FIG. 15A demonstratesthis shift in its depiction of retinal images 1534A₁ and 1534A₂, whichare representative of images of the virtual object 1530 formed on theretina 1503 of the user's eye 1502 at the first and second points intime, respectively. As such, when the mixed reality system 1500Aswitches from using depth plane 1522 to using depth plane 1523, thevirtual object 1530 may appear to “jump,” “pop,” or otherwise rapidlyshift locations within the user's FOV. FIG. 15A illustrates this shiftas image jump 1536. In particular, FIG. 15A illustrates how retinalimage 1534A₂ may be shifted up and to the right from retinal image1534A₁, and thus the user may perceive the virtual object 1530 jumpingalong jump 1536 during the depth plane switch.

Similarly, FIG. 15B shows a mixed reality system 1500B that is operatingin a blended vari-focus mode to present content to a user's eye 1502 inthe presence of a CoP misalignment. The mixed reality system 1500B may,for example, have the same architecture as the mixed reality system1500A as described above with reference to FIG. 15A. In the particularexample depicted in FIG. 15B, the mixed reality system 1500C operates inblended vari-focus mode (e.g., in a manner similar to that having beendescribed above with reference to FIGS. 14C-14D) to simultaneouslyproject virtual object 1530 onto depth planes 1522 and 1523 so as toconvert virtual object 1530 into virtual pixel sets 1532B and 1533B,respectively. In this way, the intensities of virtual pixel sets 1522Band 1533B may be blended so as to generate an accommodation cue betweendepth planes 1522 and 1523. The mixed reality system 1500B may generateand project a digitized light field representative of the virtual pixelsets 1532B and 1533B through the eyepiece 1510 and onto the retina 1503of the user's eye 1502 to form retinal image 1534B. For reasons similarto those described above with reference to FIG. 15A (e.g., parallaxshift caused by CoP misalignment), virtual pixel sets 1532B and 1533Bare not properly aligned with one another relative to the user's eye1502 and, as such, the virtual object 1530 is distorted/obscured withvisual artifacts in the retinal image 1534B. In particular, FIG. 15Billustrates how virtual content 1530 was supposed to have beenperceived, as intended image 1538, and how the CoP misalignment createsfirst and second image artifacts 1540A and 1540B (e.g., the stippledregions in retinal image 1534B).

Similarly, FIG. 15C shows a mixed reality system 1500C that is operatingin blended vari-focus mode to present content to a user's eye 1502 inthe presence of a CoP misalignment. The mixed reality system 1500C may,for example, have the same architecture as the mixed reality system1500A and/or 1500B as described above with reference to FIGS. 15A-15B.In the particular example depicted in FIG. 15C, the mixed reality system1500C operates in blended vari-focus mode (e.g., in a manner similar tothat having been described above with reference to FIGS. 14E-14F) tosimultaneously project one portion of virtual object 1530 onto depthplane 1522 and project another portion of virtual object 1530 onto depthplane 1523. As such, the portion of virtual object 1530 that isprojected onto depth plane 1522 is converted into virtual pixel set1532C, while the portion of virtual object 1530 that is projected ontodepth plane 1523 is converted into virtual pixel set 1533C. In this way,virtual pixel sets 1532C and 1533C may provide different focus cues. Themixed reality system 1500C may generate and project a digitized lightfield representative of the virtual pixel sets 1532C and 1533C throughthe eyepiece 1510 and onto the retina 1503 of the user's eye 1502 toform retinal image 1534C. For reasons similar to those described abovewith reference to FIGS. 15A and 15B (e.g., parallax shift caused by CoPmisalignment), virtual pixel sets 1532C and 1533C are not properlyaligned with one another relative to the user's eye 1502 and, as such,the virtual object 1530 appears fractured (e.g., separated at the seamsbetween the two corresponding portions of the virtual object 1530) inthe retinal image 1534C. In particular, the virtual objection 1530 mayhave a dislocation artifact 1542, as shown in retinal image 1534C.

Indeed, the correct alignment of the perspective of the virtual cameraand the perspective of the user's eye (which corresponds to thepositional/optical configuration of the user's eye) may be important toa digital light-field display device's ability to present graphics thatare of relatively high perceptual quality. In some examples, aparticular virtual camera perspective may be leveraged in a digitallight-field display device, and may correspond to a perspective in whichthe virtual camera is positioned at the center of the effective apertureof the display-plus-eye optical system.

The eye perspective position may correspond to the position of theeffective entrance pupil of the eye (generally referred to herein as theeye “CoP”), which is about 5.01 millimeters in front of the center ofcornea curvature the optical axis. In order to maintain proper alignmentbetween the pinhole of a render camera and such a location, the systemmay obtain information about the real world and the user's eyeperspective. In some examples, such information can be inferred frommeasurements of the user's eye. Such measurements may be obtained by eyetracking module 614. The position of the eye's CoP may be calculated orotherwise inferred by walking or moving to a location about 5.01millimeters from the cornea center position such as position 1008 ofFIG. 10 (toward the outer surface or cornea of the eye) along thecurrent optical or visual axis. Thus, a user's CoP may change if thewearable device moves relative to the user's face (as such, informationfrom registration observer 620 may be utilized in identifying a user'sCoP) and/or may also change as the user looks at different portions oftheir FOV (changing their optical or visual axis and thus changing theireye's CoP).

Examples of Render Camera Modes in a Mixed Reality System

As illustrated in FIGS. 16A-17B, the wearable system may utilizedifferent render camera modes including a pupil render camera mode, acenter of rotation (CoR) render camera mode, and a hybrid pupil-CoRrender camera mode. Additionally, each of these render camera modes maybe used in conjunction with each of the display focal modes describedherein in connection with FIGS. 13-15C. In at least some embodiments,the light-field render controller 618 may be configured to select whichrender camera mode to utilize at any particular time and may make suchselections based on suitable data, such as eye tracking data and, inparticular, the quality of eye tracking data. As an example, thelight-field render controller 618 or other module in the wearable systemmay select the pupil render camera mode or may bias the hybrid rendercamera mode towards the user's CoP whenever the eye tracking data isrelatively stable and the wearable system is able to identify thelocation of the CoP with low jitter or noise. In contrast, if thetracking data is limited or noisy, the light-field render controller 618may be configured to select the CoR render camera mode or bias thehybrid render camera mode toward the user's CoR. The light-field rendercontroller 618 may determine left and right camera positions inreal-time (e.g., by way of render camera controller 758) in accordancewith the selected render camera mode and may provide data indicative ofthe left and right camera positions (and the selected render cameramode) to the render engine 622.

Pupil Render Camera Mode

In the pupil render camera mode, the pinhole camera of a render camera(e.g., a simulated camera position which the render engine 622 may usein generating content for a particular user's perspective) may be slavedto the position of the estimated user's CoP for all time (e.g., asindicated by module 732 described above). In particular, the pinholecamera of a right eye render camera may be slaved to the user's righteye CoP, while the pinhole camera of a left eye render camera may beslaved to the user's left eye CoP. Thus, virtual image content presentedby the display has the perspective of the location of the CoP, which isjust in front of the pupil (e.g., within the anterior chamber of theeye).

FIGS. 16A-16B show a system 1600A operating in the pinhole cameratracked to live pupil mode (e.g., the pupil render camera mode). Thesystem 1600A includes an eyepiece 1610 through which an eye 1602 canview virtual content projected onto one or more of depth planes 1621,1622, and 1623. FIGS. 16A-16B further show a render camera 1630positioned at the CoP of eye 1602. It can be seen that eye 1602 is in afirst pose relative to eyepiece 1610 in FIG. 16A, and is in a second,different pose relative to eyepiece 1610 in FIG. 16B. As such, it canfurther be seen that the render camera 1630 is in one position in FIG.16A (e.g., a position corresponding to the CoP of eye 1602 in FIG. 16A),and is in another, different position in FIG. 16B (e.g., a positioncorresponding to the CoP of eye 1602 in FIG. 16B).

With a pinhole render camera tracked to a pupil in real-time, theabsolute position (and orientation) of the pinhole camera and therelative position (and relative orientation) between the pinhole rendercamera and the pupil changes stochastically over time. The visuals inthis mode may be jittery if the pupil position from eye tracking isnoisy and/or is not sufficiently filtered. Slaving the pinhole of therender camera to the actual position of the eye's CoP for all timeattempts to account for all pupil movement with respect to display,e.g., both low frequency changes (like slippage and IPD) as well as highfrequency changes from rotation of eye. This may introduce highfrequency dynamics into the rendering system and result in undesirabletemporal artifacts (jitter/jumping).

FIG. 16C illustrates another example in which the pinhole of a rendercamera is aligned with an eye's center of perspective or approximatelywith an eye's pupil. As depicted, a user's pupil for a single eye maymove from position A (point 164) to position B (point 1642). For anoptical eyepiece displaying a virtual object 1644, given an opticalpower, a virtual object 1644 that is meant to appear stationary willproject in 3D at either position 1646, representing a virtual objectionprojection A, or position 1648, representing a virtual object projectionB, based on the pupil position (assuming the render camera is configuredto use a pupil as the coordinate frame). Given that the projections havetwo locations by which to project, using a pupil coordinate transformedto a head coordinate will cause jitter in a stationary virtual contentas the user's eyes move. This render camera protocol may also bereferred to as a view dependent display or projection system.

Center of Rotation (CoR) Render Camera Mode

In the CoR render camera mode, the pinhole camera of a render camera(e.g., a simulated camera position which the render engine 622 may usein generating content for a particular user's perspective) may be slavedto the position of the user's center of rotation (e.g., CoR 1010 asshown in FIG. 10 ). Left and right eye centers of rotation may beestimated through execution of one or more of the operations describedabove with reference to eye tracking module 614 of FIG. 7A, such as oneor more of those described above in association with CoR estimationmodule 724. By positioning the render camera at the user's CoR, thewearable system may avoid temporal artifacts (jitter resulting from eyetracking lag in the pinhole camera tracked to live pupil mode, which maybe especially severe during microsaccade movement) in exchange forpotential spatial artifacts in the periphery (e.g., smallparallax-induced magnification/demagnification pops). This approachmakes use of the fact that relation between render camera location (eyeperspective) and what the eye is foveating on (e.g., focused on) isfixed for all the time while the glasses are stationary relative to theeyeball centers. That is, a scene may be pre-rendered in this mode suchthat the image is always perfectly correct for the fovea no matter theorientation of the eye or how it is moving, but towards the periphery isgenerally misregistered. With the render camera anchored to the CoR ofthe eye, the light field will be correctly presented at the fovea (e.g.,there is no registration to world or pop in the fovea), but may includeerrors at the periphery (e.g., as the Center of Rotation (CoR) is notthe Center of Perspective (CoP), and thus is not immune to parallaxshifts). Advantageously, such errors at the periphery may be far lessnoticeable than errors/artifacts elsewhere within an FOV, as eye acuitytapers off rapidly.

For example, FIGS. 17A-17B show a system 1700A operating in the pinholecamera fixed at CoR mode. Much like system 1600A described above withreference to FIGS. 16A-16B, the system 1700A includes an eyepiece 1710through which an eye 1702 can view virtual content projected onto one ormore of depth planes 1721, 1722, and 1723. FIGS. 17A-17B further shows arender camera 1730 positioned at the estimated CoR of eye 1702. It canbe seen that eye 1702 is in a first pose relative to eyepiece 1710 inFIG. 17A, and is in a second, different pose relative to eyepiece 1710in FIG. 17B. It can be seen that the render camera 1730 is in the sameposition in each of FIGS. 17A and 17B (e.g., the estimated CoR of theeye 1702), as the CoR does not change with changes in eye pose.

In at least some embodiments, the render camera 1730 may comprise amulti-perspective render camera 1730 which may, for example, comprise anarray of render cameras radially-distributed about the center ofrotation (CoR) at a distance (e.g., a radius) equal to the distance fromthe CoP of eye 1702 to the center of rotation of eye 1702. As such, theCoP of eye 1702 may be aligned with or nearly aligned with at least onerender camera in the array in each of several different poses. In suchembodiments, the light-field render controller 618 may select aparticular render camera from the radially-distributed array based on auser's current pose (e.g., the user's current optical axis or pupillocation), or may simply employ multiple render cameras in theradially-distributed array (or all render cameras in theradially-distributed array) simultaneously. Thus, the render controller618 may select a render camera substantially aligned and orientated witha user's CoP.

FIG. 17C illustrates another example in which the pinhole of a rendercamera is aligned with an eye's center of rotation. In some embodiments,a camera render frame is positioned at a point 1740 that encompasses allpupil positions, for example at the center of rotation of the eyeball. Avirtual object projection camera render area 1742 may be consistentregardless of the pupil position A (point 1640) and position B (point1642). The head coordinate transforms to the camera render frame. Insome embodiments, an image warp may be applied to the image to accountfor a change in eye position, but as this still renders at the sameposition, jitter is reduced or minimized. This render camera protocolmay also be referred to as a view independent display or projectionsystem.

Hybrid Pupil-CoR Render Camera Mode

In the hybrid pupil-CoR render camera mode, the pinhole camera of arender camera may be located at the pupil (CoP) position, at the CoRposition, or at any position on the line between the CoP and CoRpositions. The particular position of the render camera along that linemay vary over time, in response to changes in eye tracking data. As anexample, the light-field render controller 618 may analyze the natureand quality of the eye tracking data, as discussed in more detail below,to determine whether to locate the render camera at the user's CoP, theuser's CoR, or somewhere in-between.

In some embodiments, the system may change the location of the pinholecamera based on the determined standard deviation (or other measure ofstatistical variance) of eye tracking data. For instance, the system mayelect to position the pinhole camera at or near the center of rotation(CoR) in response to determining that the eye tracking data beingcollected is relatively noisy (and thus likely to yield substantialtemporal artifacts, such as “jitter”). Positioning the pinhole camera ator near the CoR may help to reduce jitter and other temporal artifacts.Additionally, the system may elect to position the pinhole camera at ornear the center of perspective (CoP) in response to determining that theeye tracking data being collected is relatively stable (and thus lesslikely to yield substantial temporal artifacts, such as jitter).Positioning the pinhole camera at or near the CoP may help to reduceparallax-induced (spatial) artifacts, for example as described belowwith respect to FIGS. 22-24B. In some examples, the system may simplytoggle between these two discrete locations (CoR and CoP); see Appendix(Parts I and II) for additional discussion; see also the descriptionprovided above with reference to FIGS. 10 and 15A-17B for relevantdiscussion. In other examples, the system may position the pinholecamera at any of a range of different locations along the optical axisbetween the CoR and CoP (which in some implementations may be positionedroughly 9.71 mm away from one another for an average user). In theseexamples, the system may be seen as “sliding” (or translating thelocation of) the pinhole camera along the optical axis of the eye incoordination with the determined standard deviation of eye trackingdata. When the standard deviation of eye tracking data is relativelyhigh, the system may slide the pinhole camera all the way to the CoR. Incontrast, when the standard deviation of eye tracking data is relativelylow, the system may slide the pinhole camera all the way to the CoP. Thesystem may position the pinhole of the render camera some distance infront (away from the retina) of the user's CoR in a direction along theoptical and/or visual axis. As examples, the pinhole (aperture) of therender camera may be positioned between 6.0 mm and 13.0 mm, between 7.0mm and 12.0 mm, between 8.0 mm and 11.0 mm, between 9.0 mm and 10.0 mm,between 9.5 mm and 10.0 mm, about 9.7 mm, about 9.71 mm, or othersuitable distances in front of the user's CoR.

FIGS. 18A-18D provide an example of illustrative eye tracking data andhow the system may relocate the render camera over time in response tothe eye tracking data.

Graph 1800 a of FIG. 18A provides an example of raw eye tracking datathat may be indicative of the level of noise in the eye tracking systemand the speed of movement of a user's eyes. The units of the y-axis maybe eye angular velocity in degrees per second (e.g., the change in theuser's optical axis direction in degrees per second). As shown in graph1800 a, noise and/or movement of the user's eyes may contribute tomovements in the location of a user's CoP with substantially differentvelocities over time.

Graph 1800 b of FIG. 18B provides an example of the variance of the eyetracking data of FIG. 18A. As an example, graph 1800 b may be theexponentially-weighted moving standard deviation of graph 1800 a. Ingeneral, any measure of variance of the eye tracking data may be used bythe render controller 618 in determining where to locate the rendercamera. As indicated by threshold 1810, whenever the variance (e.g.,noise, variability, velocity, etc.) in the position of the user's pupilor CoP is high, the system may position the render camera at the CoR. Incontrast and as indicated by threshold 1820, whenever the variance inthe position of the user's pupil or CoP is low, the system may positionthe render camera at the CoP. In the intermediate regions of variance,the system may position the render camera between the CoR and CoP. Asexamples, the system may place the render camera between CoR and CoP attimes A, D, and E, may place the render camera at CoR at time B, and mayplace the render camera at CoP at time C. In some embodiments, therender camera position may slide between the CoR and CoP positions as afunction of the eye tracking data (e.g., as a function of the noise orvelocity in the determined CoP data).

Graph 1800 c of FIG. 18C illustrates how the position of the rendercamera, between CoR and CoP, might vary given the example data of FIGS.18A and 18B. As shown in graph 1800 c, the render camera may beapproximately 48% of the way towards the CoP position (and away from theCoR position) along the optical axis at time A, may be located at theCoR position at time B, may be located at the CoP position at time C,may be located near the CoR position (approximately 18% away from theCoR position) at time D, and may be located near the CoP position(approximately 88% away from the CoR position) at time E.

The positions of the render camera relative to a user's eye 1802 attimes A-E are illustrated in the diagram of FIG. 18D. In particular,position D_(B) may be the CoR and may be used when the CoP location israpidly moving or noise (e.g., is undesirably jittery), position D_(D)may be just outside the CoR and may be used when the CoP location datais slightly less jittery, position D_(A) may be halfway between the CoRand CoP positions and may be used when the CoP location data is slightlyimproved, position D_(E) may be nearly to the CoP position and may beused when the CoP data is nearly sufficient to position the rendercamera at the CoP, and position D_(c) may be at the CoP and used whenthe CoP data is sufficiently stable. In general, the render camera maybe located at any point between the CoP and CoR and may be smoothlymoved between these positions over time in response to varying eyetracking data. In some implementations, the render camera position maybe adjusted between the CoP and the CoR based on the user's determinedvergence. For example, as a user's vergence or fixation depth shiftsfrom a location in space at which a depth plane is positioned to alocation in space at which a hysteresis threshold is positioned, such asthat which has been described above with reference to FIG. 12 , thesystem may shift the position of the render camera toward the CoP. Inthis way, the CoP may be utilized as the render camera location during adepth plane switch, which may be executed by the system in response todetermining that the user's vergence has passed the hysteresisthreshold, so as to reduce or minimize parallax-induced artifacts.Similarly, in such implementations, as a user's vergence or fixationdepth shifts from a location in space at which a hysteresis threshold ispositioned to a location in space at which a depth plane is positioned,the system may shift the position of the render camera toward the CoR.As described above, the system may adjust the focal length of a rendercamera when switching or otherwise adjusting depth planes. As such, insome examples, the system may adjust both the position and the focallength of a render camera as a function of the user's determinedvergence.

Determining Center of Rotation Based on Limbus Projections

As described above, for example with respect to at least FIG. 7A as wellas elsewhere herein, a center of rotation or ‘CoR’ may be determined fora user's eye. The center of rotation, as described herein, may indicatea point around which the user's eye rotates. Accordingly, in some caseswhen the user's eye rotates, there may be a point (e.g., the CoR) withinthe eye (e.g., eyeball) which is substantially fixed. As discussedabove, to determine a CoR, estimates of the location and orientation ofoptical axes and/or visual axes may be calculated for the eye fordifferent gaze directions using images obtained by an eye trackingcamera or other camera. The intersection of these different optical axesor visual axes may be used to estimate the location of a center ofrotation for the eye. Alternatively, a module such as described above(e.g., CoR estimation module 724) may utilize estimated eye dimensions.For example, the module may use a center of curvature of a cornea andestimate the CoR as being a particular distance along an optical axisfrom the center of curvature towards a retina. In some cases, thisparticular distance may be about 4.7 mm. Other methods of determiningestimates of the CoR may possibly be employed.

As discussed above, in various implementations, an eye's CoR may inform,for example, rendering, and presentation, of virtual content to theuser. As an example, a rendering engine (e.g., engine 622) may generatevirtual content via simulations of cameras positioned at the user's eyes(e.g., as if the camera located at the user's eye generated the virtualcontent) so that the virtual content is in proper perspective whenviewed by the user with the display. To determine these positions, eacheye's determined CoR may be utilized. For example, a particular cameramay be simulated as a pinhole camera with the pinhole disposed at ornear the position of a determined CoR or at a location determined usingthe location of the CoR. Thus, increasing the accuracy of a determinedCoR may provide for technical advantages with respect to the presenting,and correspondingly viewing, of virtual content.

As referenced above, a variety of methods may be utilized to determinean estimate of the location of the center of rotation of the eye. Aswill be described in more detail below, with respect to FIGS. 19-21D,for example, a limbus of a user's eye may be utilized to accuratelydetermine the eye's CoR. As an example of a limbus, FIG. 5 illustratescurve 512 a which indicates a limbic boundary of an example eye 500.Advantageously, the limbic boundary may be rapidly apparent in images ofan eye. For example, a boundary of curve 512 a may be identifiedaccording to edge detection techniques. Since a color associated withthe iris may be substantially distinct from that of the sclera, theboundary may be identified due to sudden differences at the curve 512 a.As another example, techniques such as RANSAC, machine learningtechniques, and so on, may be utilized to determine the curve 512 acorresponding to the boundary of the limbus. Additional examples ofidentifying a limbus, such as identifying a limbic boundary, aredescribed in U.S. Patent Pub. 2018/0018515, which is incorporated byreference herein.

In certain implementations, an ellipse may be projected onto an image ofa user's limbus (hereinafter referred to as a ‘projection ellipse’). Theprojection ellipse may thus represent an ellipse that is fit to theboundary of the user's limbus. As described herein, image preprocessingmodule 710 may obtain images of a user's eye for analysis. Theseobtained images may thus be utilized to determine respective projectionellipses. Since each image may include a representation of the eye in aunique orientation (e.g., the gaze may be in a different direction), theprojection ellipse may, as an example, be accordingly distinct betweensuccessive images of the eye. As will be described, for example at leastin FIG. 21A-21D, one or more projection ellipses determined fromsuccessive images may be used. The difference in the ellipses may beuseful in determining (e.g., estimate) the eye's CoR.

As an example, a cone such as a cone of rays may be projected asextending from a camera point (e.g., a pinhole camera as describedabove) through a projection ellipse associated with an image of a user'seye. See, e.g., FIG. 19 as well as FIGS. 21A-21C. Circularcross-sections of the cone may be identified as being associated withthe image. An example of these circular cross-sections 1906, 1908, isillustrated in FIG. 19 . In some implementations, the circularcross-sections may be determined using an eigenvalue/eigenvectordecomposition of the cone although other approaches to identifying thecircular cross sections may be used. In some implementations, however,vectors that provide locations of the centers of the circularcross-sections may, for example, be determined.

One of the circular cross-sections may then be selected for the image.For example, one of the circular cross-sections may correspond to a gazeof the user's eye (e.g., towards virtual content). A vector that isnormal to the selected circular cross-section (hereinafter referred toas a ‘normal vector’) may be determined. For example, the normal vectormay be normal to the vector also provides a location of the center ofthe selected circular cross-section.

In the above-described example, two or more images (e.g., successiveimages) of the eye may be analyzed. Each image of the eye, as describedabove, may represent the eye in a distinct orientation (e.g., distincteye pose). A circular cross-section may be selected for each image, andcompared to determine the CoR. For example, the CoR may be determined asan intersection of the normal vectors determined for the images. As anexample with respect to FIG. 21D, the CoR may be point 2122. In someembodiments, each normal vector may correspond to an optical axis of theeye as determined from a respective image. Thus, and as described inFIG. 7A above, the CoR may be determined as the intersection of opticalaxes determined for different eye poses.

FIG. 19 illustrates a graphical representation of a projection ellipse1902 according to the techniques described herein. As described above, acamera may be simulated as being a pinhole camera located at camerapoint 1910. The camera point 1910 may thus represent an origin of acoordinate system of one or more eye tracking cameras. An image of auser's eye may be obtained (e.g., via module 710), and an image plane1904 corresponding to the image identified. The image plane 1904 maythus include an imaged representation of the user's eye. A projectionellipse 1902 corresponding to a boundary of the user's limbus may bedetermined on the image plane 1904.

A cone 1912 formed via extending rays 1912A-D from the camera point 1910through the boundary of the projection ellipse 1902 may then beidentified. In various implementations, circular cross-sections may bedetermined along a length of the cone 1912. In the example shown, thecircular cross-sections have perimeters intersecting with and bounded bythe cone of rays. The circular cross-sections, as will be described withreference to FIG. 20 , may be uniquely determined based on a specificradius of the circular cross-sections. For example, a first circularcross-section 1906 and a second circular cross-section 1908 with a sameradius are illustrated. In some embodiments, and as will be describedbelow with respect to refining a CoR, the limbus radius may not beknown, assumed, or taken into account, when determining atwo-dimensional CoR. As an example, the radius may represent an averageradius of a limbus across a multitude of users, a radius determined forthe user based on images of the user's limbus, or may represent anarbitrary constant value. An eigenvector decomposition may optionally beperformed, and a principle axis 1914 of the cone 1912 may be identified.As illustrated in FIG. 19 , the principle axis 1914 is illustrated asextending through a geometric center of the projection ellipse 1902 tothe camera point 1910.

A first vector 1916 may be determined which passes through a center ofthe first circular cross-section 1906. Similarly, a second vector 1918may be determined which passes through the second circular cross-section1908. For examples, these vectors 1916-1918 may be determined based onthe eigenvector decomposition of the cone 1912. The circularcross-sections 1906, 1908, may be disambiguated. Thus, one of thecircular cross-sections may be identified as better corresponding to agaze associated with the imaged eye. For example, while either of thecircular cross-sections 1906, 1908, may be mathematically allowable fora given radius, one of the circular cross-sections may correspond to anactual gaze of the eye. Thus, one of the circular cross-sections may beselected for utilization in determining an estimate of the eye's CoR.

As will be described in more detail below, with respect to FIG. 20 , anormal vector that is normal to the selected circular cross-section maybe determined. For example, the normal vector may be normal to eitherthe first circular cross-section 1906 or the second circularcross-section 1908. Normal vectors may be determined for a number ofimages. An intersection point in three-dimensional space between thenormal vectors may then be identified as corresponding to the eye's CoR.An example of an intersection point is illustrated in FIG. 21D, withrespect to intersection point 2122. In some cases, a region andassociated location where the normal vectors come close together istaken as an estimate of the CoR, for example, if the normal vectors donot intersect at one point. In some implementations, a least-squaresapproach may be taken to obtain an estimate of such a point ofintersection. In such implementations, the system may identify alocation at which the sum of the squared distances to the normal vectorsis reduced or minimized as the point of intersection.

Advantageously, in some embodiments, one or more techniques may beutilized to refine the estimated CoR described above. For example, athree-dimensional location of the CoR may be refined. In this example,the refined CoR may be a more accurate representation of the eye's CoR.As an example, to refine the CoR, a plane (herein referred to as a‘projection plane’) defined by the first vector 1916 and second vector1918 may be identified. See, e.g., FIG. 19 . This projection plane mayinclude the principle axis 1914 of the cone 1912. Thus, the projectionplane may include (e.g., pass through) the geometric center of theprojection ellipse 1902. Additionally, the projection plane may include(e.g., pass through) the camera point 1910. As will be described, theprojection plane may further include (e.g., pass through) the eye's CoR.

Two or more projection planes may be utilized. For example, theseprojection planes may be determined from successive images of the user'seye. In this example, an intersection between the two or more projectionplanes may be identified. Since, as described above, each projectionplane may include the camera point 1910, the resulting line, formed fromthe intersection, may thus extend from the camera point 1910. Anintersection of the resulting line with the image plane 1904 maytherefore represent a two-dimensional location of the CoR on the imageplane 1904. To refine the CoR (e.g., the three-dimensional CoR describedabove), an intersection of the resulting line with the normal vectorsmay be identified. The intersection may thus be assigned as the refinedCoR. Optionally, the refined CoR may be identified as a point along theresulting line based on the point's proximity to the intersection orconvergence of the normal vectors. For example, a point on the resultingline which is closest (e.g., according to a root mean squared process)to the normal vectors, or an intersection or convergence thereof, may beassigned as the refined CoR. As mentioned above, in some examples, aleast-squares method may also be employed to estimate of such a point.

FIG. 20 is a flowchart of an example process 2000 for determining aneye's center of rotation (CoR) based on the eye's limbus. Forconvenience, the process 2000 will be described as being performed by adisplay system of one or more processors or processing elements.

At block 2002, the display system obtains an image of a user's eye. Asdescribed above, with respect to at least FIG. 7A, the display systemmay obtain images of a user's eye via one or more cameras or otherimaging systems.

At block 2004, the display system determines a projection ellipseforming a cone based on a limbus of the user's eye. To determine theprojection ellipse, the limbus of the user's eye may be identified inthe obtained image (e.g. limbic boundary 512 illustrated in FIG. 5 ).Without being constrained by theory, the limbic boundary may beidentified according to different techniques. As an example, an edgedetection scheme may be utilized to identify an edge between the scleraand iris. In this example, the edge may represent the limbic boundary.As another example, a machine learning model may be trained to label alimbic boundary (e.g., a neural network). The projection ellipse may insome cases be an ellipse which the display system determinesapproximates (e.g., substantially approximates) the limbic boundary.

However, and as illustrated in FIG. 21A, the obtained image may notinclude the entirety of the limbus. For example, an eyelid of the user'seye may occlude a portion of the limbus. In the example of FIG. 21A, theupper and lower portion of the limbus are occluded. The display systemmay thus estimate the projection ellipse (e.g., the semi-major andsemi-minor axis). Optionally, to estimate the projection ellipse thedisplay system may fit the ellipse based on the visible portions of thelimbus. For example, the display system may identify a semi-major axisand semi-minor axis based on the boundary of the limbus visible in theimage. Optionally, to estimate the projection ellipse the display systemmay utilize average information associated with a limbus. For example,if the left and right most portions of the limbic boundary are visible,the display system may utilize average ratios between the semi-major andsemi-minor axes of an average limbus. Optionally, the display system mayutilize previously obtained images of the user's limbus to identify thedimensions of the limbus.

As described in FIG. 19 , rays may be extended from a camera point(e.g., a pinhole camera associated with an imaging system that obtainedthe image, aperture of camera, etc.) through the projection ellipseboundary. For example, FIG. 21A illustrates 12 example rays beingextended from the camera point through respective locations along theprojection ellipse boundary. These extended rays may thus form a cone.In some embodiments, the cone may be an elliptical cone. Acircular-cross section of the cone may be selected (e.g., as describedabove) and a normal vector identified based on the circular-crosssection. Different techniques to identify such normal vectors, and/orcircular cross-sections, may be employed. An example of a technique isdescribed in more detail below.

Without subscribing to any particular scientific or mathematicaltheories, in some implementations, the projection ellipse describedabove may be described according to the following equation:ax ² +bxy+cy ² dx+ey+f=0in which the five coefficients (e.g., a, b, c, d, e, f) are defined asfollows:a=A ² sin² ϕ+B ² cos²ϕb=2(B ² −A ²)sin ϕ cos ϕc=A ² cos² ϕ+B ² sin²ϕd=−2ax _(c) −by _(c)e=−bx _(c)−2cy _(c)f=ax _(c) ² +bx _(c) y _(c) +cy _(c) ² −A ² B ²

The projection ellipse equation described above may be written accordingto homogenous coordinates {tilde over (x)}=(x y 1)^(T) as:{tilde over (x)}=C _(2D) {tilde over (x)}=0

The conic matrix, C_(2D), identified above may be defined as:

$C_{2D} = \begin{pmatrix}\alpha & {b/2} & {d/2} \\{b/2} & c & {e/2} \\{d/2} & {e/2} & f\end{pmatrix}$

The display system may adjust the above-identified equation according tointrinsic camera parameters (e.g., an intrinsic camera parametermatrix). Thus, the conic equation may be represented as:C=Λ ^(T) C _(2D)Λ

At block 2006, the display system determines vectors associated with aselected circular cross-section of the cone. As described in block 2004,the cone formed by rays extending from the camera point through theboundary of the projection ellipse may be determined. In someimplementations, to determine the vectors associated with the circularcross-sections, the display system may determine eigenvalues andeigenvectors of the cone. For example, the cone ‘C’ may be decomposedinto an orthogonal matrix and a diagonal matrix:C=UDU ^(T)

The display system may select an eigenvalue ‘λ₃’ (e.g., from diagonalmatrix ‘D’) with a sign opposite that of the remaining two eigenvalues.The corresponding eigenvector may then be determined as corresponding toa principle axis of the cone (e.g., principle axis 1914 illustrated inFIG. 19 ). The principle axis may be represented as being based on thecorresponding eigenvector (referred to below as e₃):ē ₃=sign[(ē ₃)_(z) ]·ē ₃

The display system may identify, from the two remaining eigenvalues, asmallest eigenvalue according to absolute values of the two remainingeigenvalues. This smallest eigenvalue may be referred to as ‘λ₂’, andthe remaining eigenvalue may be referred to as ‘λ₁’

The display system may then determine the following:

$l_{1} = \sqrt{\frac{{❘\lambda_{3}❘} \cdot \left( {{❘\lambda_{1}❘} - {❘\lambda_{2}❘}} \right)}{{❘\lambda_{1}❘} \cdot \left( {{❘\lambda_{1}❘} + {❘\lambda_{3}❘}} \right)}}$$l_{2} = \sqrt{\frac{{❘\lambda_{1}❘} \cdot \left( {{❘\lambda_{2}❘} + {❘\lambda_{3}❘}} \right)}{{❘\lambda_{3}❘} \cdot \left( {{❘\lambda_{1}❘} + {❘\lambda_{3}❘}} \right)}}$$l_{3} = \sqrt{\frac{{❘\lambda_{1}❘} - {❘\lambda_{2}❘}}{{❘\lambda_{1}❘} + {❘\lambda_{3}❘}}}$$l_{4} = \sqrt{\frac{{❘\lambda_{2}❘} + {❘\lambda_{3}❘}}{{❘\lambda_{1}❘} + {❘\lambda_{3}❘}}}$

As described in FIG. 19 , two circular cross-sections of the cone may bedetermined. For example, in some implementations, the specific twocircular cross-sections may be determined based on specified radius. Theradius, may, as an example represent a radius associated with a limbus(e.g., as described above).

In some implementations, the circular cross-sections may be defined, atleast in part, according to vectors based on the eigenvectordecomposition described above. The vectors may represent vectorsextending through a center of each of the circular cross-sections. Forexample, vector c₁ and c₂ may be determined, along with correspondingnormal vectors n₁ and n₂.c ₁ =R(−l ₁ ē ₁ +l ₂ ē ₃) n ₁ =l ₃ ē ₁ +l ₄ ē ₃c ₂ =R(l ₁ ē ₁ +l ₂ ē ₃) n ₂ =−l ₃ ē ₁ +l ₄ ē ₃

The display system selects one of the circular-cross sections. Forexample, the display system may select one of the vector pairs [c₁, n₁]or [c₂, n₂] which corresponds to a gaze of the eye. In this example, thedisplay system may determine which associated circular cross-sectioncorresponds to the user looking at a virtual display (e.g., virtualcontent). As an example, the display system may utilize thecorresponding normal vectors to identify which normal vector points atvirtual content. Thus, the disambiguation may be performed using thedisplay system. In some embodiments, other schemes such as utilizing atwo-dimensional CoR (e.g., as described below) may be utilized to selectfrom among the vector pairs. Other methods, however, may be used todetermine any one or more of the projection ellipse, the cone, thecircular cross-sections through the cone, the normal vectors for thecircular cross-sections, or to select a cross-section and/or associatednormal vector. Variations of the mathematical methods discussed abovemay be used or such mathematical methods need not be used. A variety ofother methods may be employed.

As will be described below with respect to block 2008, the displaysystem may utilize the selected vector pair to determine the eye's CoR.For example, the display system may utilize the selected vector pair incombination with other vector pairs, selected from one or moresuccessive images, to determine the CoR.

At block 2008, the display system determines the eye's CoR. As describedin blocks 2002-2006 above, the display system may determine a normalvector based on the user's limbus as represented in an image of theuser's eye. For example, the normal vector may represent an optical axisof the user's eye. As described in block 2006, the display system maydetermine a vector pair such as for example a vector pair [c_(x),n_(x)]. To determine the eye's CoR, the display system may utilize twoor more normal vectors determined from respective images of the user'seye. For example, the two or more images may be successive images of theuser's eye (e.g., obtained according to a certain periodicity). Asanother example, the two or more images may represent images taken athreshold time apart. In the example of FIG. 20 , the display system mayoptionally perform block 2008 upon receipt of a threshold number ofimages. For example, the image described in blocks 2002-2006 mayrepresent a last image of the threshold number of images. Otherapproaches are possible. Advantageously, the display system mayperiodically refine, and update, the determined CoR as will be describedbelow.

Similar to the above discussion, for each of the images the displaysystem may determine a vector providing a location of a center of aselected circular cross-section. The display system may also determine anormal vector that is normal to the selected circular cross-section. Thenormal vector may, as an example, represent an optical axis of the eye.For example, the normal vectors may identify the varying optical axes ofthe eye according to eye poses represented in the images.

To determine the CoR, the display system may identify a location (e.g.,three-dimensional location) at which the normal vectors intersect,converge, or are in close proximity, for example, most of the vectorsintersect, converge, or are in close proximity or on average the vectorsintersect, converge, or are in close proximity. For example, a root meansquared process may be employed. Thus, and as described in FIG. 7A, theidentified location may represent a location at which the optical axesintersect. This intersection point may be assigned as the eye's CoR. Insome implementations, the distance from the center of the limbus circleto the CoR may be determined and stored for future use. In at least someof these implementations, the system may store such a distance inassociation with a particular user, and later rely upon the storeddistance to determine the CoR for that particular user (e.g., byidentifying a position along the normal vector that is the storeddistance away from the center of the limbus). Similarly, in someembodiments, the CoR may be determined by identifying a position alongthe normal vector that is an assumed distance away from the center ofthe limbus. In these embodiments, such an assumed distance maycorrespond to a population average or other predefined value.

As new images of the user's eye are received, the display system mayrefine the CoR. For example, the display system may determine one ormore new CoRs based on respective images (e.g., respective groups of athreshold number of images). The display system may then refine the CoRbased on the new CoRs. As an example, the display system may compute aroot mean square of the CoRs. Optionally, the display system may utilizeall obtained images, or a threshold number, to continually update theCoR. For example, the display system may initially utilize two or moreimages to determine the CoR. As new images are received, the displaysystem may perform the process 2000 with all, or a threshold number, ofthe received images. With the increase in the number of images, and thusincrease in a number of determined normal vectors, the accuracy of theCoR may be increased. Other approaches are possible.

FIG. 21A illustrates a first projection ellipse 2100 determined based ona first gaze of a user's eye. As described above, an image of the user'seye may be obtained. Based on the image, a boundary of the eye's limbusmay be determined. As illustrated, a first projection ellipse 2100 hasbeen determined in an image plane 2112 based on the boundary of thelimbus. In the illustrated example, points along the boundary areidentified. Rays forming a cone 2104 are illustrated as extending from acamera point 2102 through the points along the boundary. A circularcross-section 2106 has been determined, for example, as described inFIG. 20 , with a center 2108 of the circular cross-section 2106identified in FIG. 21A.

As described in FIGS. 19-20 , two circles from the cone 2104 may bedetermined for a given radius. In the example of FIG. 21A-21D, one ofthe circles may be selected. For example, and as described above, one ofthe circles may correspond to a gaze pointing at a virtual display beingviewed by a user (e.g., virtual content). Extending from the center 2108of the circle is a vector 2110. The vector may thus represent one of thenormal vectors n₁ or n₂ described above with respect to FIG. 20 .

FIG. 21B illustrates a second projection ellipse 2114 determined basedon a second gaze of the user's eye. In this example, an image subsequentto the image described in FIG. 21A is obtained. Similar to the above,the second projection ellipse 2114 is identified in image plane 2112.Additionally, vector 2116 has been determined.

FIG. 21C illustrates a third projection ellipse 2118 determined based ona third gaze of the user's eye. In this example, an image subsequent tothe images described in FIGS. 21A-21B is obtained. Similar to the above,the third projection ellipse 2118 is identified in image plane 2112.Additionally, vector 2120 has been determined.

FIG. 21D illustrates estimating a center of rotation (CoR) based on thedetermined vectors 2110, 2116, 2120. The display system may utilize thedetermined vectors to identify a point at which the vectors intersect.For example, the display system has determined that the vectorsintersect at point 2122. To determine intersection point 2122, thedisplay system may optionally utilize known physical information. Forexample, a distance between camera point 2102 and the user's eye may beemployed. Alternatively, or in addition, the intersection or locationwhere the normals appear to come in close proximity or converge, forexample, where most of the vectors intersect, converge, or are in closeproximity or on average the vectors intersect, converge, or are in closeproximity, may be determined. The display system may then assignintersection point 2122 or the intersection or location where thenormals appear to come in close proximity or converge, for example,where most of the vectors intersect, converge, or are in close proximityor on average the vectors intersect, converge, or are in closeproximity, as the eye's CoR.

As described above, with respect to FIGS. 19-21D, images of a user's eyemay be utilized to determine an estimate of the eye's CoR. Additionalvariations and techniques may be employed and fall within the scope ofthe disclosure. As an example, the display system may determine theeye's CoR based on images of a user's eye during rotation and/ormovement of the eye. Through analyzing variations in geometric aspectsof the eye the display system may identify the CoR. For example, thedisplay system may identify optical axes of the eye as represented inthe images. The display system may utilize the optical axes to determinethe CoR. Thus, additional techniques and variations may be employedwhich utilize geometric aspects of a user's eye to determine an estimateof the eye's CoR.

As an example, the display system may determine an array of positionsassociated with an image of a user's eye. The array of positions maycorrespond to spatial locations on the image. Example spatial locationsmay be associated with a portion of the user's eye (e.g., the limbus,pupil, iris, and so on). Thus, in some implementations, the array ofpositions may be fit to an extremity of the portion. The extremity ofthe portion may be associated with a curve (e.g., the projection ellipsedescribed above), which is determined for the portion of the user's eye.In some implementations, the display system may identify linear pathsextending from a same point through the array of positions. As describedabove, the linear paths (e.g., rays) may extend from a camera point. Thelinear paths may form a cone, and a particular circular cross-section ofthe cone may be selected. For example, the particular cross-section mayhave a certain radius (e.g., a radius of an average limbus or of theuser's limbus, a radius of an average pupil or the of the user's pupil,and so on). A vector normal to the particular cross-section may beidentified. The display system may determine an intersection of multipleof the normal vectors, and then assign the intersection as the CoR. Insome implementations, one or more of the techniques for determining 3Dpoints of intersection between two or more optical axes and/or othervectors, as described above with reference to FIG. 7A, may be leveragedto determine 3D points of intersection between the normal vectorsdescribed herein with reference to FIGS. 19-21D. Similarly, in someembodiments, one or more of the techniques for averaging and/or applyinga filter (e.g., a Kalman-type filter) to a plurality of estimated CoRpositions or other 3D positions, as described above with reference toFIG. 7A, may be leveraged to determine and/or refine the CoR estimatesdescribed herein with reference to FIGS. 19-21D. Variations and othermethods may be employed.

Refining Center of Rotation (CoR) Utilizing Additional ExampleTechniques

As described in FIGS. 20-21 , the display system may determine an eye'sCoR as a location (e.g., three-dimensional location) at which multiplevectors intersect, converge, or are in close proximity (e.g., vectors2110, 2116, 2120). Alternatively, or additionally, the display systemmay refine the determined CoR according to one or more other techniques.For example, and as described in FIGS. 19-20 , vectors c₁ and c₂ (e.g.,vectors 1916, 1918) may be determined from a cone (e.g., cone 1912)formed via rays extending through a projection ellipse (e.g., ellipse1902). These vectors may form a projection plane. Similar to the aboveregarding utilizing a plurality of images, the display system maydetermine projection planes for two or more images.

To refine the determined CoR, the display system may utilize thedetermined projection planes. For example, in some implementations, thedisplay system may determine intersections of the projection planes(e.g., in three-dimensional space). The intersection of the projectionplanes may, as an example, result in a line. Additionally, this line maypass through the camera point (e.g., point 1910) from which the coneassociated with the projection planes extend. To refine the CoR, thedisplay system may identify a point in the image plane (e.g., imageplane 1904) at which this resulting line intersects. The display systemmay then assign this point as the two-dimensional CoR on the imageplane. As described in FIG. 19 , the display system may adjust thetwo-dimensional CoR along the resulting line (e.g., in three-dimensionalspace). For example, the refined CoR may be assigned as a location alongthe resulting line close or closest to an intersection of the normalvectors (e.g., vectors 2110, 2116, 2120).

Optionally, the display system may refine the eye's CoR according to thefollowing technique. Different images of the user's eye (e.g.,successive images) may be obtained. For each image, the display systemmay determine an intersection of the vectors c₁ and c₂ with the imageplane (e.g., image plane 1904). The display system may then connect aline between (1) the intersection of vector c₁ with the image plane and(2) the intersection of vector c₂ with the image plane. Thus, the linemay represent the intersection of the projection plane, as defined byvectors c₁ and c₂, with the image plane. The images may thus beassociated with respective lines, and the display system may determine atwo-dimensional CoR based on proximity of points in the image plane tothe lines. For example, the display system may perform a root meansquared (RMS) process (e.g., based on random sample consensus) todetermine the two-dimensional CoR. Similar to the above, the displaysystem may refine the CoR (e.g., as described in FIGS. 20-21 ) byadjusting the two-dimensional CoR along a line connecting thetwo-dimensional CoR and camera point.

With respect to the above example, the display system may optionallyselect one of the vectors c₁ or c₂ for each of the images. For example,the display system may select one of the vectors c₁ or c₂ based on agaze of an eye. As another example, the display may select one of thevectors c₁ or c₂ based on the respective vectors intersection with theimage plane. In this example, the vector which intersects the imageplane closer to that of the two-dimensional CoR (e.g., as describedabove), may be selected. The display system may then determine a pointat which each selected vector intersects with the image plane. Similarto the above, the display system may then determine the two-dimensionalCoR based on proximity of points in the image plane to the determinedintersection points. For example, an RMS process may be utilized.

Determining Center of Rotation (CoR) Based on Pupil Projections

Similar to the above description in FIGS. 19-21D related to utilizing alimbus of an eye, a pupil may alternatively, or additionally, beutilized to determine an eye's CoR. For example, a projection ellipsemay be determined for a pupil included in an image of a user's eye.Since the pupil is located behind a surface of the user's cornea, lightmay be refracted at the surface. However, the main effect of suchrefraction may be a rotation of a normal vector extending from the pupilwith respect to the un-refracted pupil. The refraction effect of thecornea may be considered as discussed herein to determine the refractionbased location of the pupil. As a non-limiting example, the refractionbased location of the pupil center may also coincide with the projectionplane as determined according to the techniques described above. Thedisplay system may utilize images of the user's pupil to determinerefraction planes as described herein. Thus, the display system mayutilize images of the user's pupil to determine projection planes asdescribed herein. For example, a CoR ray (or “eyeball center ray”) maybe found the same way for the pupil as was described above for thelimbus. Including the refraction effect for pupil may, as an example,not change the CoR ray calculation. Then, using a fixed (e.g., average)pupil radius, the display system may find circular sections and theircorresponding normal (e.g., as described above).

The display system may then determine the eye's CoR based on the pupilin similar ways as described above for determining the CoR based on thelimbus. For example, an ellipse may be fit to the pupil image. A cone ofrays may be projected through the ellipse. Circular cross-sections maybe fit to the cone. Normals through a plurality of cross-sections obtainusing a plurality of images may be used may considered and theintersection or convergence of those normal may be identified. Anestimate of the CoR may be obtained therefrom. Other methods, includingother methods described herein, such as using projection planes may beemployed to determine an estimate of the CoR.

Center of Perspective (CoP) Analysis/Derivation

In the pinhole camera model, the center of perspective (CoP) may beconsidered as an aperture of a pinhole camera. An example property ofthis CoP is that it may also be the origin of the object angle space.Thus, and as an example with reference to this origin, objects which areat a same angle from a pinhole will map to the same pixel (e.g.,overlap). As another example, if a scene is rigidly rotated about theCoP, then a projected image will translate and the objects in the scenewill not experience parallax shifts.

In developing the systems and techniques described herein, twohypotheses were developed and tested. The first hypothesis was that thecenter of the principal plane is the CoP, and the second hypothesis wasthat the center of the aperture is the CoP. The principal plane, as anexample, may represent a plane where incident light rays may beconsidered to bend due to refraction. As will be described, it may bedetermined that the center of aperture is the CoP. For example, FIGS.22, 23A-23B, and 24A-24B, along with their corresponding descriptionsbelow, may be informative as to the analysis associated with the testingof these two hypotheses.

FIG. 22 illustrates an optical system 2200 including two point lightsources 2201 and 2202, an aperture 2204, a lens 2206, and a projectionscreen 2208. In this example, the aperture 2204, lens 2206, andprojection screen 2208 of optical system 2200 may be representative ofor at least functionally similar to anatomical features of the humaneye. For instance, the aperture 2204, lens 2206, and projection screen2208 of optical system 2200 may correspond to a pupil, lens, and retina,respectively.

In FIG. 22 , rays of light emitted by the first point light source 2201are represented as solid lines extending from the first point lightsource 2201 toward the projection screen 2208, while rays of lightemitted by the second point light source 2202 are represented as dottedlines extending from the second point light source 2202 toward theprojection screen 2208. Some of the light emitted from each of the twopoint light sources 2201 and 2202 may propagate through the aperture2204 and the lens 2206 and ultimately strike the projection screen 2208to form an incident ray pattern 2210.

FIG. 23A illustrates an optical system 2300 in a first stage (e.g.,stage “A”), which includes two point light sources 2301 and 2302, anaperture 2304, a lens 2306, and a projection screen 2308. Optical system2300 may, for example, be architecturally and functionally identical orsimilar to optical system 2200, as described in further detail abovewith reference to FIG. 22 . More specifically, optical system 2300 mayrepresent a particular implementation of optical system 2200, asdescribed in further detail above with reference to FIG. 22 , in whichthe center of the principal plane is defined as the CoP such that theabovementioned first hypothesis may be tested. Since the lens 2306 mayserve to define the principal plane in optical system 2300, it followsthat the CoP in this example may correspond to lens center 2307.

An evaluation of optical system 2300 for the two abovementionedimplications may be informative as to whether the abovementioned firsthypothesis is true or false. That is, in order for the abovementionedfirst hypothesis to be true, the image that is projected onto theprojection screen 2308 (e.g., the incident ray pattern) should notundergo any parallax shifts when the two point light sources 2301 and2302 are rigidly rotated about the lens center 2307 (e.g., rotated alongradii of curvature 2311 and 2312, respectively), but merely experience atranslation. Referring again to FIG. 22 , this means that the image thatis projected onto the projection screen 2308 should look like atranslated version of the incident ray pattern 2210 (e.g., image isformed at a slightly different location on the projection screen 2308)when the two point light sources 2301 and 2302 are rigidly rotated toany pair of points along radii of curvature 2311 and 2312, respectively.

FIG. 23B illustrates optical system 2300 in a second stage (e.g., stage“B”) in which the two point light sources 2301 and 2302 have beenrigidly rotated about lens center 2307 along radii of curvature 2311 and2312, respectively, to become rigidly rotated point light sources 2301′and 2302′. It can be seen that some of the light emitted from each ofthe two rigidly rotated point light sources 2301′ and 2302′ propagatesthrough the aperture 2304 and the lens 2306 and ultimately strikes theprojection screen 2308 to form an incident ray pattern 2310.

However, upon examination of the incident ray pattern 2310, it can alsobe seen that the relative position (e.g., on the projection screen 2310)between rays of the incident ray pattern 2310 that originate from thefirst rigidly rotated light source 2301′ and rays of the incident raypattern 2310 that originate from the second rigidly rotated light source2302′ has shifted. Referring again to FIG. 22 , it can be further notedthat the incident ray pattern 2310 yielded in FIG. 23B does not appearto resemble a translated version of the incident ray pattern 2210. Inother words, rigid rotation of the two point light sources 2301 and 2302about the lens center 2307 resulted in a parallax shift. For thisreason, the lens center 2307 may be seen as an unsuitable location forthe CoP. Thus, FIGS. 22, 23A, and 23B and the accompanying analysisthereof can be as effectively demonstrating that the abovementionedfirst hypothesis is false.

FIG. 24A illustrates an optical system 2400 in a first stage (e.g.,stage “A”), which includes two point light sources 2401 and 2402, anaperture 2404, a lens 2406, and a projection screen 2408. Optical system2400 may, for example, be architecturally and functionally identical orsimilar to optical system 2200, as described in further detail abovewith reference to FIG. 22 , in much the same way as described above withreference to FIGS. 23A and 23B. More specifically, optical system 2400may represent a particular implementation of optical system 2200, asdescribed in further detail above with reference to FIG. 22 , in whichthe aperture center 2407 is defined as the CoP such that theabovementioned second hypothesis may be tested.

An evaluation of optical system 2400 similar to evaluation of opticalsystem 2300, as described above with reference to FIGS. 23A and 23B, maybe informative as to whether the abovementioned second hypothesis istrue or false. That is, in order for the abovementioned secondhypothesis to be true, the image that is projected onto the projectionscreen 2408 (e.g., the incident ray pattern) should not undergo anyparallax shifts when the two point light sources 2401 and 2402 arerigidly rotated about the aperture center 2407 (e.g., rotated alongradii of curvature 2411 and 2412, respectively), but merely experience atranslation.

FIG. 24B illustrates optical system 2400 in a second stage (e.g., stage“B”) in which the two point light sources 2401 and 2402 have beenrigidly rotated about aperture center 2407 along radii of curvature 2411and 2412, respectively, to become rigidly rotated point light sources2401′ and 2402′. It may be appreciated that light from the two rigidlyrotated point light sources 2401′ and 2402′ strikes the projectionscreen 2408 to form an incident ray pattern 2410 that appears toresemble a translated version of the incident ray pattern 2210, asdescribed above with reference to FIG. 22 . That is, although theposition of the incident ray pattern 2410 with respect to the projectionscreen 2408 may differ from that of the incident ray pattern 2210 withrespect to the projection screen 2208, the relative position (e.g., onthe projection screen 2310) between rays of the incident ray pattern2410 that originate from the first rigidly rotated light source 2401′and rays of the incident ray pattern 2410 that originate from the secondrigidly rotated light source 2402′ does not appear to have shifted. Assuch, rigid rotation of the two point light sources 2401 and 2402 aboutthe aperture center 2407 does not appear to yield a parallax shift.Given that the two abovementioned implications appear to have been metin this example, the aperture center 2407 may be seen as a suitablelocation for the CoP. Thus, FIGS. 22, 24A, and 24B and the accompanyinganalysis thereof can be as effectively demonstrating that theabovementioned second hypothesis is true.

As such, it may, in some embodiments, be desirable to align the CoP inthe Render World (e.g., location of the pinhole of a render camera) witha portion of a user's eye (in the Real World) which is the anatomicalequivalent of the aperture center 2407. Because the human eye furtherincludes a cornea (which imparts additional optical power to lightpropagating toward the retina), the anatomical equivalent of theaperture center 2407 may not correspond to the center of the pupil oriris of a user's eye, but may instead correspond to a region of theuser's eye positioned between the outer surface of the cornea of theuser's eye and the center of the pupil or iris of the user's eye. Forexample, the anatomical equivalent of the aperture center 2407 maycorrespond to a region within the anterior chamber of the user's eye.

Examples for when Eye Tracking is Unavailable

In some embodiments, eye tracking may not be provided or may betemporarily unavailable. As examples, the eye tracking camera 324 orlight sources 326 may be obscured, damaged, or disabled by a user, theenvironmental lighting conditions may make eye tracking prohibitivelydifficult, the wearable system may be improperly fitted in a manner thatprevents eye tracking, the user may be squinting or have eyes that arenot easily tracked, etc. At such times, the wearable system may beconfigured to fall back upon various strategies for positioning therender camera and selecting depth planes in the absence of eye trackingdata.

With respect to the render camera, the wearable system may position therender camera to a default position if the user's pupils are notdetected for longer than a predetermined threshold, such as a fewseconds or longer than a typical blink. The wearable system may move therender camera to the default position in a smooth movement, which mayfollow an over-damped oscillator model. The default position may bedetermined as part of a calibration process of the wearable system to aparticular user. The default position may be a user's left and righteyes' centers of rotation. These are merely illustrative examples.

With respect to the depth plane, the wearable system may provideaccommodation cues based on the depth of virtual content, as opposed tothe vergence depth of the user as previously discussed. In someembodiments, the wearable system may receive, obtain, or determineinformation estimating where the user is likely to be looking and mayprovide matching accommodation cues. As an example, the wearable systemmay be displaying content that the user is likely to be focused on, suchas a video clip, may assume that the user is looking at the content, andmay provide the content on a depth plane (or with blended depth planes)to provide accommodation cues that match the depth of that content.

Computer Vision to Detect Objects in Ambient Environment

As discussed above, the display system may be configured to detectobjects in or properties of the environment surrounding the user. Thedetection may be accomplished using a variety of techniques, includingvarious environmental sensors (e.g., cameras, audio sensors, temperaturesensors, etc.), as discussed herein.

In some embodiments, objects present in the environment may be detectedusing computer vision techniques. For example, as disclosed herein, thedisplay system's forward-facing camera may be configured to image theambient environment and the display system may be configured to performimage analysis on the images to determine the presence of objects in theambient environment. The display system may analyze the images acquiredby the outward-facing imaging system to perform scene reconstruction,event detection, video tracking, object recognition, object poseestimation, learning, indexing, motion estimation, or image restoration,etc. As other examples, the display system may be configured to performface and/or eye recognition to determine the presence and location offaces and/or human eyes in the user's field of view. One or morecomputer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

One or more of these computer vision techniques may also be usedtogether with data acquired from other environmental sensors (such as,e.g., microphone) to detect and determine various properties of theobjects detected by the sensors.

As discussed herein, the objects in the ambient environment may bedetected based on one or more criteria. When the display system detectsthe presence or absence of the criteria in the ambient environment usinga computer vision algorithm or using data received from one or moresensor assemblies (which may or may not be part of the display system),the display system may then signal the presence of the object.

Machine Learning

A variety of machine learning algorithms may be used to learn toidentify the presence of objects in the ambient environment. Oncetrained, the machine learning algorithms may be stored by the displaysystem. Some examples of machine learning algorithms may includesupervised or non-supervised machine learning algorithms, includingregression algorithms (such as, for example, Ordinary Least SquaresRegression), instance-based algorithms (such as, for example, LearningVector Quantization), decision tree algorithms (such as, for example,classification and regression trees), Bayesian algorithms (such as, forexample, Naive Bayes), clustering algorithms (such as, for example,k-means clustering), association rule learning algorithms (such as, forexample, a-priori algorithms), artificial neural network algorithms(such as, for example, Perceptron), deep learning algorithms (such as,for example, Deep Boltzmann Machine, or deep neural network),dimensionality reduction algorithms (such as, for example, PrincipalComponent Analysis), ensemble algorithms (such as, for example, StackedGeneralization), and/or other machine learning algorithms. In someembodiments, individual models may be customized for individual datasets. For example, the wearable device may generate or store a basemodel. The base model may be used as a starting point to generateadditional models specific to a data type (e.g., a particular user), adata set (e.g., a set of additional images obtained), conditionalsituations, or other variations. In some embodiments, the display systemmay be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

The criteria for detecting an object may include one or more thresholdconditions. If the analysis of the data acquired by the environmentalsensor indicates that a threshold condition is passed, the displaysystem may provide a signal indicating the detection the presence of theobject in the ambient environment. The threshold condition may involve aquantitative and/or qualitative measure. For example, the thresholdcondition may include a score or a percentage associated with thelikelihood of the reflection and/or object being present in theenvironment. The display system may compare the score calculated fromthe environmental sensor's data with the threshold score. If the scoreis higher than the threshold level, the display system may detect thepresence of the reflection and/or object. In some other embodiments, thedisplay system may signal the presence of the object in the environmentif the score is lower than the threshold. In some embodiments, thethreshold condition may be determined based on the user's emotionalstate and/or the user's interactions with the ambient environment.

In some embodiments, the threshold conditions, the machine learningalgorithms, or the computer vision algorithms may be specialized for aspecific context. For example, in a diagnostic context, the computervision algorithm may be specialized to detect certain responses to thestimulus. As another example, the display system may execute facialrecognition algorithms and/or event tracing algorithms to sense theuser's reaction to a stimulus, as discussed herein.

It will be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. For example,computing systems may include general purpose computers (e.g., servers)programmed with specific computer instructions or special purposecomputers, special purpose circuitry, and so forth. A code module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming language.In some embodiments, particular operations and methods may be performedby circuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

Other Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

EXAMPLES

of a user to display virtual image content in a vision field of saiduser are described herein such as the examples enumerated below:

Example 1: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field at different amounts of atleast one of divergence and collimation and thus the displayed virtualimage content appears to originate from different depths at differentperiods of time; one or more eye tracking cameras configured to imagethe user's eye; and processing electronics in communication with thedisplay and the one or more eye tracking cameras, the processingelectronics configured to obtain an estimate of a center of rotation ofsaid eye based on images of said eye obtained with said one or more eyetracking cameras.

Example 2: The display system of Example 1, further comprising one ormore light sources disposed on said frame with respect to said user'seye to illuminate said user's eye, said one or more eye tracking camerasforming images of said eye using said light from said one or more lightsources.

Example 3: The display system of Example 1 or 2, wherein said one ormore light sources comprises at least two light sources disposed on saidframe with respect to said user's eye to illuminate said user's eye.

Example 4: The display system of Example 1 or 3, wherein said one ormore light sources comprises infrared light emitters.

Example 5: The display system of any of the Examples 1 to 4, wherein oneor more light sources form one or more glints on said eye and saidprocessing electronics is configured to determine a location of saidcornea based on said one or more glints.

Example 6: The display system of any of the Examples 1 to 5, whereinsaid cornea has associated therewith a cornea sphere having a center ofcurvature and said processing electronics is configured to determine alocation of said center of curvature of said cornea sphere.

Example 7: The display system of Example 5, wherein said cornea hasassociated therewith a cornea sphere having a center of curvature andsaid processing electronics is configured to determine a location ofsaid center of curvature of said cornea sphere based on said one or moreglints.

Example 8: The display system of any of the Examples above, wherein saidone or more eye tracking camera is configured to image said pupil ofsaid eye.

Example 9: The display system of any of the Examples above, wherein saidprocessing electronics is configured to determine the location of saidcenter of said pupil.

Example 10: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine at least aportion of a boundary between said iris and said pupil.

Example 11: The display system of Example 10, wherein said processingelectronics is configured to determine a center of said boundary betweensaid iris and said pupil.

Example 12: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of said pupil in three-dimensional space with respect to acenter of curvature of said cornea.

Example 13: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of said optical axis.

Example 14: The display system of Example 12, wherein said processingelectronics is configured to determine a location and orientation ofsaid optical axis based on a location of said center of said pupil inthree-dimensional space.

Example 15: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine said location andorientation of said optical axis based on a location of said center ofsaid pupil in three-dimensional space with respect to a center ofcurvature of said cornea.

Example 16: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea.

Example 17: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea and a location and orientation of said optical axis.

Example 18: The display system of Example 17, wherein said processingelectronics is configured to determine the location of said center ofrotation of said eye by translating a particular distance along saidoptical axis from said center of curvature of said cornea.

Example 19: The display system of Example 18, wherein said particulardistance from said center of curvature to said center of rotation isbetween 4.0 mm and 6.0 mm.

Example 20: The display system of Example 18 or 19, wherein saidparticular distance from said center of curvature to said center ofrotation is about 4.7 mm.

Example 21: The display system of Example 18 or 19, wherein saidparticular distance is fixed.

Example 22: The display system of Example 18 or 19, wherein saidprocessing electronics is configured to determine the particulardistance based at least on one or more images of said eye previouslyobtained with said one or more eye tracking cameras.

Example 23: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis, offset from said optical axis, based onsaid location and orientation of said optical axis.

Example 24: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis based on an angular rotation with respectto said optical axis.

Example 25: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis based on an angular rotation of between4.0° and 6.5° with respect to said optical axis.

Example 26: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of a visual axis based on an angular rotation of about 5.2°with respect to said optical axis.

Example 27: The display system of any of the Examples above, whereinsaid processing electronics are configured to determine a location andorientation of a visual axis based at least on one or more images ofsaid eye previously obtained with said one or more eye tracking cameras.

Example 28: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a center ofrotation of said eye based multiple determinations of said location ofsaid optical axis or visual axis over a period of time during which saideye is rotating.

Example 29: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine said center ofrotation by identifying a region of intersection of multipledeterminations of said location of said optical axis or a visual axisover a period of time during which said eye is rotating.

Example 30: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance of said user where left and right eyes of a user are gazingbased on a determination of the location and orientation of said opticalaxes for said left and right eyes of the user.

Example 31: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance of said user where left and right eyes of a user are gazingbased on a determination of the location and orientation of said visualaxes for said left and right eyes of the user.

Example 32: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance where left and right eyes of a user are gazing based onidentifying a region of intersection of said visual axes for said leftand right eyes of the user.

Example 33: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a vergencedistance where left and right eyes of a user are gazing by projectingthe visual axes for said left and right eyes onto a horizontal plane andidentifying a region of intersection of said projections of the visualaxes for said left and right eyes onto a horizontal plane.

Example 34: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the relativeamounts of at least one of divergence, and collimation to project imagecontent based on a determination of said vergence distance.

Example 35: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 36: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 37: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 38: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 39: The display system of any of the Examples above, whereinsaid head-mounted display receives light from a portion of theenvironment in front of the user at a first amount of divergence andtransmits the light from the portion of the environment in front of theuser to the user's eye with a second amount of divergence that issubstantially similar to the first amount of divergence.

Example 40: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain the estimate of thecenter of rotation by filtering, averaging, applying a Kalman filter, orany combinations thereof a plurality of estimated center of rotationpositions.

Example 41: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that is rendered as if captured by a camerahaving an aperture at the determined position of the center of rotationof said user's eye.

Example 42: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera at saidcenter of rotation to render virtual images to be presented to said eye.

Example 43: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render cameraconfigured to render virtual images to be presented to said eye that arerendered as if captured by a camera having an aperture closer to saidthe center of rotation than said retina of said eye.

Example 44: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render cameraconfigured to render virtual images to be presented to said eye that arerendered as if captured by a camera having an aperture at said thecenter of rotation of said eye.

Example 45: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera at saidcenter of rotation to render virtual images to be presented to said eye,said render camera modeled with an aperture at said center of rotationof said eye.

Example 46: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field at different amounts of atleast one of divergence and collimation and thus the displayed virtualimage content appears to originate from different depths at differentperiods of time; one or more eye tracking cameras configured to imagethe user's eye; and processing electronics in communication with thedisplay and the one or more eye tracking cameras, the processingelectronics configured to obtain a position estimate of a center ofperspective of said eye based on images of said eye obtained with saidone or more eye tracking cameras, said center of perspective beingestimated to be proximal said pupil of said eye or between said corneaand said pupil of said eye, wherein said processing electronics isconfigured to present said virtual image content to said user's eye thatare rendered by a render camera located at said center of perspective.

Example 47: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than saidretina.

Example 48: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture closer to said center of perspective than a center ofrotation of the eye.

Example 49: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture at said center of perspective.

Example 50: The display system of any of the Examples above, whereinsaid center of perspective is not located at said pupil of said eye.

Example 51: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain an estimate of saiduser's eye pose over time and wherein the processing electronics adjustthe position of the render camera based at least in part upon the user'seye pose.

Example 52: The display system of any of the Examples above, wherein theprocessing electronics is configured to track said user's eye pose overtime and wherein the position of the render camera is adjusted over timein response to changes in said user's eye pose over time.

Example 53: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain the estimate of thecenter of perspective by filtering a plurality of estimated center ofperspective positions.

Example 54: The display system of any of the Examples above, wherein theprocessing electronics is configured to obtain the estimate of thecenter of perspective by averaging and/or applying a Kalman filter to aplurality of estimated center of perspective positions.

Example 55: The display system of any of the Examples above, wherein thecenter of perspective comprises a position within the anterior chamberof said user's eye.

Example 56: The display system of any of the Examples above, wherein thecenter of perspective comprises a position in front of said pupil ofsaid user's eye.

Example 57: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 1.0 mm and2.0 mm in front of said pupil of said user's eye.

Example 58: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is about 1.0 mm in frontof said pupil of said user's eye.

Example 59: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 0.25 mm and1.0 mm in front of said pupil of said user's eye.

Example 60: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 0.5 mm and1.0 mm in front of said pupil of said user's eye.

Example 61: The display system of any of the Examples above, wherein thecenter of perspective comprises a position that is between 0.25 mm and0.5 mm in front of said pupil of said user's eye.

Example 62: The display system of any of the Examples above, wherein thecenter of perspective lies along the optical axis of said eye andwherein said processing electronics are further configured to obtain theposition estimate of the center of perspective by obtaining a positionestimate of the optical axis of said eye.

Example 63: The display system of any of the Examples above, wherein thecenter of perspective lies along the optical axis of said eye at aposition between an outer surface of the cornea and the pupil of saideye and wherein said processing electronics are further configured toobtain the position estimate of the center of perspective by obtaining aposition estimate of the optical axis of said eye.

Example 64: The display system of any of the Examples above, wherein thecenter of perspective lies along the optical axis of said eye at aposition between an outer surface of the cornea and the pupil of saideye and wherein said processing electronics are further configured toobtain the position estimate of the center of perspective by obtaining aposition estimate of the optical axis of said eye and a positionestimate of a center of rotation of said eye, the cornea of said eye,the iris of said eye, the retina of said eye, and the pupil of said eyeor any combinations thereof.

Example 65: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 66: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 67: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 68: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 69: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or more eyetracking cameras capturing images of said eye using said light from saidone or more light sources.

Example 70: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 71: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least three light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 72: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 73: The display system of any of the Examples above, whereinsaid one or more light sources form one or more glints on said eye andsaid processing electronics is configured to determine the position ofthe center of curvature of said cornea based on said one or more glints.

Example 74: The display system of any of the Examples above, whereinsaid one or more light sources form one or more glints on said eye andsaid processing electronics is configured to determine athree-dimensional position of the center of curvature of said corneabased on said one or more glints.

Example 75: The display system of any of the Examples above, whereinsaid one or more eye-tracking cameras are further configured to imagesaid pupil of the user's eye and wherein said processing electronics arefurther configured to determine the position of said pupil of said eyebased at least on the image of said pupil from the one or moreeye-tracking cameras.

Example 76: The display system of any of the Examples above, whereinsaid one or more eye-tracking cameras are further configured to imagesaid pupil of the user's eye and wherein said processing electronics arefurther configured to determine a three-dimensional position of saidpupil of said eye based at least on the image of said pupil from the oneor more eye-tracking cameras.

Example 77: The display system of any of the Examples above, whereinsaid one or more eye-tracking cameras are further configured to imagesaid pupil of the user's eye and wherein said processing electronics arefurther configured to determine the position of said pupil of said eyebased on the position of the center of curvature of said cornea andbased on the image of said pupil from the one or more eye-trackingcameras.

Example 78: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the optical axisof said eye based on the three-dimensional position of the center ofcurvature of said cornea and based on the three-dimensional position ofsaid pupil.

Example 79: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a visual axis ofsaid eye based on the optical axis.

Example 80: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the visual axisof said eye based on the optical axis and the three-dimensional positionof at least one of the center of curvature of said cornea, said pupil orboth.

Example 81: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine athree-dimensional position of the center of rotation of said eye basedon the three-dimensional position of the center of curvature of saidcornea.

Example 82: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine athree-dimensional position of the center of rotation of said eye basedon the three-dimensional position of the center of curvature of saidcornea and based on said optical axis.

Example 83: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the distancebetween said eye and the additional eye of said user based at least onthe three-dimensional position of the center of rotation of said eye.

Example 84: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine an interpupillarydistance between said eye and the additional eye of said user based atleast on the three-dimensional position of the center of rotation ofsaid eye.

Example 85: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the optical axis of said eye.

Example 86: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the optical axis of said eye andon a determined optical axis of the additional eye of said user.

Example 87: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the vergencedistance of said user based at least on the visual axis of said eye andon a determined visual axis of the additional eye of said user.

Example 88: The display system of any of the Examples above, whereinsaid display is configured to project collimated light into said user'seye.

Example 89: The display system of any of the Examples above, whereinsaid display is configured to project collimated light corresponding toan image pixel into said user's eye at a first period of time anddivergent light corresponding to said image pixel into said user's eyeat a second period of time.

Example 90: The display system of any of the Examples above, whereinsaid display is configured to project light corresponding to an imagepixel having a first amount of divergence into said user's eye at afirst period of time and to project light corresponding to said imagepixel having a second amount of divergence, greater than the firstamount of divergence, into said user's eye at a second period of time.

Example 91: A method of rendering virtual image content in a displaysystem configured to project light to an eye of a user to display thevirtual image content in a vision field of said user, said eye having acornea, an iris, a pupil, a lens, a retina, and an optical axisextending through said lens, pupil, and cornea, said method comprising:with one or more eye tracking cameras configured to image said eye ofthe user to track movements of said eye, determining a position of acenter of rotation of said eye; with a render engine, rendering virtualimage content with a render camera at said center of rotation of saideye, said render camera configured to render virtual images to bepresented to said eye; and with a head-mounted display, projecting lightinto said user's eye to display the rendered virtual image content tothe user's vision field at different amounts of divergence such that thevirtual image content appears to originate from different depths atdifferent periods of time.

Example 92: The method of any of the Examples above, wherein said rendercamera is configured to render virtual images to be presented to saideye that are rendered as if captured by a camera having an aperturecloser to said the center of rotation than said retina of said eye.

Example 93: The method of any of the Examples above, wherein said rendercamera is configured to render virtual images to be presented to saideye that are rendered as if captured by a camera having an aperture atsaid the center of rotation.

Example 94: The method of any of the Examples above, wherein said rendercamera is modeled with an aperture at said center of rotation of saideye.

Example 95: The method of any of the Examples above, wherein said rendercamera is modeled with an aperture, a lens, and a detector.

Example 96: The method of any of the Examples above, wherein said rendercamera has an aperture at a position along a line between (i) thedetermined position of the center of rotation of said eye and (ii) thedetermined position of the at least one of said iris or pupil.

Example 97: The method of any of the Examples above, further comprising:with the one or more eye tracking cameras, determining a position of acenter of perspective of said user's eye, wherein the center ofperspective of said user's eye is located less than approximately 1.0 mmfrom the pupil of said user's eye; and with the render engine, renderingthe virtual image content with the render camera, wherein said rendercamera has an aperture at the determined position of the center ofperspective of said user's eye.

Example 98: The method of any of the Examples above, further comprising:with the render engine, rendering the virtual image content with therender camera, wherein said render camera has an aperture at a positionalong a line between (i) the determined position of the center ofrotation of said eye and (ii) the determined position of the center ofperspective of said user's eye.

Example 99: The method of any of the Examples above, further comprising:with processing electronics in communication with the one or more eyetracking cameras, determining a measure of change with time of thedetermined position of the center of perspective of said user's eye; andwith the processing electronics, if it is determined that the measure ofchange with time exceeds a first threshold, directing the render engineto render the virtual content with the render camera, wherein the rendercamera has an aperture at the determined position of the center ofrotation of said eye.

Example 100: The method of any of the Examples above, furthercomprising: with the processing electronics, if it is determined thatthe measure of change with time is below a second threshold, directingthe render engine to render the virtual content with the render camera,wherein the render camera has an aperture at the determined position ofthe center of perspective of said eye, and wherein the first thresholdis indicative of a higher level change with time in the determinedposition of the center of perspective of said user's eye than the secondthreshold.

Example 101: The method of any of the Examples above, furthercomprising: with the processing electronics, if it is determined thatthe measure of change with time is below a second threshold, directingthe render engine to render the virtual content with the render camera,wherein the render camera has an aperture at the determined position ofthe center of perspective of said eye.

Example 102: The method of any of the Examples above, furthercomprising: with the processing electronics, if it is determined thatthe measure of change with time is between the first and secondthresholds, directing the render engine to render the virtual contentwith the render camera, wherein the render camera has an aperture at apoint along a line between (i) the determined position of the center ofrotation of said eye and (ii) the determined position of the center ofperspective of said eye.

Example 103: The method of any of the Examples above, furthercomprising: with at least a portion of said display, said portion beingtransparent and disposed at a location in front of the user's eye whenthe user wears said head-mounted display, transmitting light from aportion of the environment in front of the user and said head-mounteddisplay to the user's eye to provide a view of said portion of theenvironment in front of the user and said head-mounted display.

Example 104: The method of any of the Examples above, furthercomprising: with the one or more eye tracking cameras, determining aposition of at least one of said iris, pupil, or lens

Example 105: The method of any of the Examples above, furthercomprising: with the render engine, rendering the virtual image contentwith the render camera, said render camera configured to present virtualimages to said eye images that are rendered as if captured by a camerahaving an aperture at a position along a line between (i) the determinedposition of the center of rotation of said eye and (ii) the determinedposition of the at least one of said iris or pupil.

Example 106: The method of any of the Examples above, furthercomprising: with the one or more eye tracking cameras, determining aposition of a center of perspective of said user's eye, wherein thecenter of perspective of said user's eye is located less thanapproximately 1.0 mm from the pupil of said user's eye; and with therender engine, rendering the virtual image content with the rendercamera, said render camera configured to present virtual images to saideye images that are rendered as if captured by a camera having anaperture at the determined position of the center of perspective of saiduser's eye.

Example 107: The method of any of the Examples above, furthercomprising: with the render engine, rendering the virtual image contentwith the render camera, said render camera configured to present virtualimages to said eye images that are rendered as if captured by a camerahaving an aperture at a position along a line between (i) the determinedposition of the center of rotation of said eye and (ii) the determinedposition of the center of perspective of said user's eye.

Example 108: The method of any of the Examples above, furthercomprising: with processing electronics in communication with the one ormore eye tracking cameras, determining a measure of change with time ofthe determined position of the center of perspective of said user's eye;and with the processing electronics, if it is determined that themeasure of change with time exceeds a first threshold, directing therender engine to render the virtual content with the render camera as ifcaptured by a camera having an aperture at the determined position ofthe center of rotation of said eye.

Example 109: The method of any of the Examples above, furthercomprising: with the processing electronics, if it is determined thatthe measure of change with time is below a second threshold, directingthe render engine to render the virtual content with the render cameraas if captured by a camera having an aperture at the determined positionof the center of perspective of said eye, wherein the first threshold isindicative of a higher level change with time in the determined positionof the center of perspective of said user's eye than the secondthreshold.

Example 110: The method of any of the Examples above, furthercomprising: with the processing electronics, if it is determined thatthe measure of change with time is below a second threshold, directingthe render engine to render the virtual content with the render cameraas if captured by a camera having an aperture at the determined positionof the center of perspective of said eye.

Example 111: The method of any of the Examples above, furthercomprising: with the processing electronics, if it is determined thatthe measure of change with time is between the first and secondthresholds, directing the render engine to render the virtual contentwith the render camera as if captured by a camera having an aperture ata point along a line between (i) the determined position of the centerof rotation of said eye and (ii) the determined position of the centerof perspective of said eye.

Example 112: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, and a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field at different amounts ofdivergence and thus the displayed virtual image content appears tooriginate from different depths at different periods of time, whereinsaid head-mounted display is configured to project light into saiduser's eye having a first amount of divergence at a first period of timeand is configured to project light into said user's eye having a secondamount of divergence at a second period of time, wherein the firstamount of divergence is different from the second amount of divergence;one or more eye tracking cameras configured to image the user's eye; andprocessing electronics in communication with the display and the one ormore eye tracking cameras, the processing electronics configured toobtain an estimate of a center of rotation of said eye based on imagesof said eye obtained with said one or more eye tracking cameras, obtainan estimate of a vergence distance of said user based on images of saideye obtained with said one or more eye tracking cameras, and shift fromprojecting light into said user's eye at the first amount of divergenceto projecting light into said user's eye at the second amount ofdivergence based on the estimated vergence distance of said user.

Example 113: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 114: The display system of any of the Examples above, whereinsaid processing electronics is further configured to, based on images ofsaid eye obtained with said one or more eye tracking cameras, detect ablink of said eye.

Example 115: The display system of any of the Examples above, whereinsaid processing electronics is further configured to, based on images ofsaid eye obtained with said one or more eye tracking cameras, detect asaccade of said eye.

Example 116: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence based onthe determined vergence distance of said user and based on whether theprocessing electronics have detected the blink of said eye.

Example 117: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence based onthe determined vergence distance of said user and based on whether theprocessing electronics have detected the saccade of said eye.

Example 118: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence based onthe determined vergence distance of said user and based on whether theprocessing electronics have detected at least one of the saccade or theblink of said eye.

Example 119: The display system of any of the Examples above, whereinsaid first amount of divergence is associated with vergence distances ina first range and wherein said second amount of divergence is associatedwith vergence distances in a second range.

Example 120: The display system of any of the Examples above, whereinsaid first amount of divergence is associated with vergence distances ina first range, wherein said second amount of divergence is associatedwith vergence distances in a second range and wherein the first andsecond ranges overlap but are not equal.

Example 121: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the firstrange and lies within the second range.

Example 122: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the secondrange and lies within the first range.

Example 123: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the firstrange and lies within the second range and also detecting a blink ofsaid eye.

Example 124: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user lies outside the firstrange and lies within the second range and also detecting a saccade ofsaid eye.

Example 125: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user has been outside thefirst range and within the second range for longer than a predeterminedperiod of time.

Example 126: The display system of any of the Examples above, whereinsaid processing electronics is configured to shift from projecting lightinto said user's eye at the first amount of divergence to projectinglight into said user's eye at the second amount of divergence upondetermining the vergence distance of said user has been outside thefirst range and within the second range for longer than a predeterminedperiod of time of at least 10 seconds.

Example 127: The display system of any of the Examples above, whereinsaid head-mounted display comprises a first display element configuredto project light having the first amount of divergence and a seconddisplay element configured to project light having the second amount ofdivergence.

Example 128: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a discrete display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using only one of the first display element.

Example 129: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a blended display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using both of the first and second display elementsfor each of the frames.

Example 130: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a blended display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using both of the first and second display elementsfor each of the frames and wherein, in the blended display mode, thedisplay is configured to project light, using the first and seconddisplay elements, that is perceived by a user as having a given amountof divergence that is between the first and second amounts ofdivergence.

Example 131: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content in a multi-focus display mode in which thedisplay is configured to project light associated with a plurality ofsequential frames using both of the first and second display elementsfor each of the frames, wherein, in the multi-focus display mode, thedisplay is configured to project light associated with first virtualimage content at a third amount of divergence and to project lightassociated with second virtual image content at a fourth amount ofdivergence, and wherein the third amount of divergence is different fromthe fourth amount of divergence.

Example 132: The display system of any of the Examples above, whereinthird and fourth amounts of divergence are each between the first andsecond amounts of divergence.

Example 133: The display system of any of the Examples above, wherein atleast one of the third and fourth amounts of divergence are between thefirst and second amounts of divergence.

Example 134: The display system of any of the Examples above, whereinthe third and fourth amounts of divergence are respectively equal to thefirst and second amounts of divergence.

Example 135: The display system of any of the Examples above, whereinthe display is configured to project light associated with the firstvirtual image in a first region of the user's vision field and toproject light associated with the second virtual image in a secondregion of the user's vision field and wherein the first and secondregions are different.

Example 136: The display system of any of the Examples above, whereinthe display is configured to project light associated with the firstvirtual image in a first region of the user's vision field and toproject light associated with the second virtual image in a secondregion of the user's vision field and wherein the first and secondregions do not overlap.

Example 137: A display system configured to project light to left andright eyes of a user to display virtual image content in a vision fieldof said user, each of said eyes having a cornea, an iris, a pupil, alens, a retina, and an optical axis extending through said lens, pupil,and cornea, said display system comprising: a frame configured to besupported on a head of the user; a head-mounted display disposed on theframe, said display configured to project light into said user's leftand right eyes to display virtual image content to the user's visionfield at different amounts of at least one of divergence and collimationand thus the displayed virtual image content appears to originate fromdifferent distances from the user's left and right eyes at differentperiods of time; a first eye tracking camera configured to image theuser's left eye; a second eye tracking camera configured to image theuser's right eye; and processing electronics in communication with thedisplay and the first and second eye tracking cameras, the processingelectronics configured to obtain an estimate of an interpupillarydistance between the user's left and right eyes based on images of saidleft and right eyes obtained with said first and second eye trackingcameras.

Example 138: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or more eyetracking cameras forming images of said eye using said light from saidone or more light sources.

Example 139: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 140: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 141: The display system of any of the Examples above, whereinone or more light sources form one or more glints on said eye and saidprocessing electronics is configured to determine a location of saidcornea based on said one or more glints.

Example 142: The display system of any of the Examples above, whereinsaid cornea has associated therewith a cornea sphere having a center ofcurvature and said processing electronics is configured to determine alocation of said center of curvature of said cornea sphere.

Example 143: The display system of any of the Examples above, whereinsaid cornea has associated therewith a cornea sphere having a center ofcurvature and said processing electronics is configured to determine alocation of said center of curvature of said cornea sphere based on saidone or more glints.

Example 144: The display system of any of the Examples above, whereinsaid one or more eye tracking camera is configured to image said pupilof said eye.

Example 145: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the location ofsaid center of said pupil.

Example 146: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine at least aportion of a boundary between said iris and said pupil.

Example 147: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a center of saidboundary between said iris and said pupil.

Example 148: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of said pupil in three-dimensional space with respect to acenter of curvature of said cornea.

Example 149: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of said optical axis.

Example 150: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location andorientation of said optical axis based on a location of said center ofsaid pupil in three-dimensional space.

Example 151: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine said location andorientation of said optical axis based on a location of said center ofsaid pupil in three-dimensional space with respect to a center ofcurvature of said cornea.

Example 152: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea.

Example 153: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a location ofsaid center of rotation of said eye based on a center of curvature ofsaid cornea and a location and orientation of said optical axis.

Example 154: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine the location ofsaid center of rotation of said eye by translating a particular distancealong said optical axis from said center of curvature of said cornea.

Example 155: A method of rendering virtual image content in a displaysystem configured to project light to left and right eyes of a user todisplay the virtual image content in a vision field of said user, eachof said eyes having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, said methodcomprising: with one or more eye tracking cameras configured to imagesaid eyes of the user to track movements of said eyes, determining aposition of a center of rotation of said left eye and a position of acenter of rotation of said right eye; with processing electronics incommunication with the one or more eye tracking cameras, estimating saiduser's interpupillary distance based on the determined positions of thecenter of rotation of said left and right eyes; with the one or more eyetracking cameras, determining a current left eye pose and a currentright eye pose; and with the processing electronics, estimating saiduser's current vergence distance by comparing said estimatedinterpupillary distance and said determined current left eye pose andsaid determined current right eye pose.

Example 156: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said pupil ofsaid user's left eye and a position of said pupil of said user's righteye.

Example 157: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said corneaof said user's left eye and a position of said cornea of said user'sright eye.

Example 158: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said iris ofsaid user's left eye and a position of said iris of said user's righteye.

Example 159: The method of any of the Examples above, whereindetermining said current left and right eye poses comprises, with theone or more eye tracking cameras, estimating a position of said lens ofsaid user's left eye and a position of said lens of said user's righteye.

Example 160: The method of any of the Examples above, wherein estimatingsaid user's current vergence distance comprises: with processingelectronics, estimating a distance between said positions of said irisesof said user's left and right eyes; and with the processing electronics,estimating said user's current vergence distance based on a comparisonof said estimated interpupillary distance and said estimated distancebetween said positions of said irises of said user's left and righteyes.

Example 161: The method of any of the Examples above, furthercomprising: with a head-mounted display, projecting light into saiduser's eye to display the rendered virtual image content to the user'svision field at different amounts of divergence such that the virtualimage content appears to originate from different depths at differentperiods of time.

Example 162: The method of any of the Examples above, furthercomprising: with at least a portion of said display, said portion beingtransparent and disposed at a location in front of the user's eye whenthe user wears said head-mounted display, transmitting light from aportion of the environment in front of the user and said head-mounteddisplay to the user's eye to provide a view of said portion of theenvironment in front of the user and said head-mounted display.

Example 163: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field at different amounts of atleast one of divergence and collimation and thus the displayed virtualimage content appears to originate from different depths at differentperiods of time; one or more eye tracking cameras configured to imagethe user's eye; and processing electronics in communication with thedisplay and the one or more eye tracking cameras, the processingelectronics configured to obtain a position estimate of a center ofrotation of said eye based on images of said eye obtained with said oneor more eye tracking cameras and configured to obtain a directionestimate of the optical axis of said eye based on said images, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between6.0 mm and 13.0 mm in a direction away from said retina.

Example 164: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between7.0 mm and 12.0 mm in a direction away from said retina.

Example 165: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between8.0 mm and 11.0 mm in a direction away from said retina.

Example 166: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between9.0 mm and 10.0 mm in a direction away from said retina.

Example 167: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye by between9.5 mm and 10.0 mm in a direction away from said retina.

Example 168: The display system of any of the Examples above, whereinsaid processing electronics is configured to present said virtual imagecontent to said user's eye that are rendered as if captured by a camerahaving an aperture disposed along the optical axis and spaced apart fromthe estimated position of the center of rotation of said eye byapproximately 9.7 mm.

Example 169: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame.

Example 170: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics disposed at a location remote from said frame.

Example 171: The display system of any of the Examples above, whereinsaid processing electronics includes electronics on said frame andelectronics on a belt pack.

Example 172: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 173: The display system of any of the Examples above, furthercomprising one or more light sources disposed on said frame with respectto said user's eye to illuminate said user's eye, said one or more eyetracking cameras capturing images of said eye using said light from saidone or more light sources.

Example 174: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least two light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 175: The display system of any of the Examples above, whereinsaid one or more light sources comprises at least three light sourcesdisposed on said frame with respect to said user's eye to illuminatesaid user's eye.

Example 176: The display system of any of the Examples above, whereinsaid one or more light sources comprises infrared light emitters.

Example 177: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said eye having a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field at different amounts of atleast one of divergence and collimation and thus the displayed virtualimage content appears to originate from different depths at differentperiods of time; one or more eye tracking cameras configured to imagethe user's eye; and processing electronics in communication with thedisplay and the one or more eye tracking cameras, wherein saidprocessing electronics is configured to present said virtual imagecontent to said user's eye that are rendered by a render camera havingan aperture located at the pupil of the eye or between the pupil and thecornea of the eye.

Example 178: The display system of any of the Examples above, whereinthe aperture of the render camera is located at a position that isbetween 1.0 mm and 2.0 mm in front of said pupil of said user's eye.

Example 179: The display system of any of the Examples above, whereinthe aperture of the render camera is located a position that is about1.0 mm in front of said pupil of said user's eye.

Example 180: The display system of any of the Examples above, whereinthe aperture of the render camera is located at a position that isbetween 0.25 mm and 1.0 mm in front of said pupil of said user's eye.

Example 181: The display system of any of the Examples above, whereinthe aperture of the render camera is located at a position that isbetween 0.5 mm and 1.0 mm in front of said pupil of said user's eye.

Example 182: The display system of any of the Examples above, whereinthe aperture of the render camera is located at position that is between0.25 mm and 0.5 mm in front of said pupil of said user's eye.

Example 183: The display system of any of the Examples above, whereinthe aperture of the render camera is located at the pupil of the eye.

Example 184: The display system of any of the Examples above, whereinthe aperture of the render camera is not located at the pupil of theeye.

Example 185: Any of the claims above, wherein the camera comprises apinhole camera.

Example 186: Any of the claims above, wherein the aperture comprises apinhole of a pinhole camera.

Any of the above Examples or Additional Examples can be combined.Additionally, any of the above Examples or Additional Examples can beintegrated with a head mounted display. In addition, any of the aboveExamples or Additional Examples can be implemented with a single depthplane and/or with one or more variable depth planes (e.g., one or moreelements with variable focusing power that provide accommodation cuesthat vary over time).

Furthermore, apparatus and methods for determining a variety of values,parameters, etc., such as, but not limited to, anatomical, optical, andgeometric features, locations, and orientations, etc., are disclosedherein. Examples of such parameters include, for example, the center ofrotation of the eye, the center of curvature of the cornea, the centerof the pupil, the boundary of the pupil, the center of the iris, theboundary of the iris, the boundary of the limbus, the optical axis ofthe eye, the visual axis of the eye, the center of perspective, but arenot limited to these. Determinations of such values, parameters, etc.,as recited herein include estimations thereof and need not necessarilycoincide precisely with the actual values. For example, determinationsof the center of rotation of the eye, the center of curvature of thecornea, the center or boundary of the pupil or iris, the boundary of thelimbus, the optical axis of the eye, the visual axis of the eye, thecenter of perspective, etc., may be estimations, approximations, orvalues close to, but not the same as, the actual (e.g., anatomical,optical, or geometric) values or parameters. In some cases, for example,root mean square estimation techniques are used to obtain estimates ofsuch values. As an example, certain techniques described herein relateto identifying a location or point at which rays or vectors intersect.Such rays or vectors, however, may not intersect. In this example, thelocation or point may be estimated. For example, the location or pointmay be determined based on root mean square, or other, estimationtechniques (e.g., the location or point may be estimated to be close toor the closest to the rays or vectors). Other processes may also be usedto estimate, approximate or otherwise provide a value that may notcoincide with the actual value. Accordingly, the term determining andestimating, or determined and estimated, are used interchangeablyherein. Reference to such determined values may therefore includeestimates, approximations, or values close to the actual value.Accordingly, reference to determining a parameter or value above, orelsewhere herein should not be limited precisely to the actual value butmay include estimations, approximations or values close thereto.

What is claimed is:
 1. A display system configured to project light toan eye of a user to display virtual image content in a vision field ofsaid user, said eye having a cornea, an iris, a pupil, a lens, a retina,and an optical axis extending through said lens, pupil, and cornea, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field, at least a portion of saiddisplay being transparent and disposed at a location in front of theuser's eye when the user wears the frame such that said transparentportion transmits light from a portion of the environment in front ofthe user and said head-mounted display to the user's eye to provide aview of said portion of the environment in front of the user and saidhead-mounted display; an inward-facing imaging system configured toimage the user's eye; and processing electronics in communication withthe inward-facing imaging system, the processing electronics configuredto obtain an estimate of a three-dimensional (3D) position of a centerof rotation (CoR) of said eye based at least partly on a plurality ofsaid eye images obtained with said inward-facing imaging system, whereinobtaining the estimate of the 3D position of the CoR includes:determining a plurality of potential 3D positions of the CoR, whereineach potential 3D position of the CoR is determined as a particulardistance from a center of an estimated corneal sphere that includes thecornea of the eye; calculating a variation for each of the potential 3Dpositions of the CoR, wherein the variation is calculated based on ananalysis of multiple images of the user's eye captured over a period oftime; and determining the estimate of the 3D position of the CoR as thepotential 3D position that exhibits the lowest variation among theplurality of potential 3D positions.
 2. The display system of claim 1,wherein the plurality of potential 3D positions of the CoR includes aninitial value of the distance from the center of an estimated cornealsphere.
 3. The display system of claim 2, wherein the initial value isan average of experimentally measured distance values.
 4. The displaysystem of claim 2, wherein the initial value is 4.7 mm.
 5. The displaysystem of claim 1, wherein the processing electronics are furtherconfigured to determine an estimate of the corneal sphere based onanalyzing the plurality of said eye images.
 6. The display system ofclaim 1, wherein the CoR is the estimated position about which the eyerotates relative to at least two axes.
 7. The display system of claim 1,wherein obtaining the estimate of the 3D position of the CoR isperformed during an initial user calibration session of the displaysystem.
 8. A method implemented by a display system configured toproject light to an eye of a user to display virtual image content in avision field of said user, said eye having a cornea, an iris, a pupil, alens, a retina, and an optical axis extending through said lens, pupil,and cornea, said method comprising: displaying, via the display system,virtual image content to the user's vision field; with an inward-facingimaging system configured to image said eye of the user to trackmovements of said eye, obtaining an estimate of a center of rotation ofsaid eye based on a plurality of said eye images obtained with saidinward-facing imaging system; and determining an estimate of athree-dimensional (3D) position of a center of rotation (CoR) of saideye based at least partly on a plurality of said eye images obtainedwith said inward-facing imaging system, wherein obtaining the estimateof the 3D position of the CoR includes: determining a plurality ofpotential 3D positions of the CoR, wherein each potential 3D position ofthe CoR is determined as a particular distance from a center of anestimated corneal sphere that includes the cornea of the eye;calculating a variation for each of the potential 3D positions of theCoR, wherein the variation is calculated based on an analysis ofmultiple images of the user's eye captured over a period of time; anddetermining the estimate of the 3D position of the CoR as the potential3D position that exhibits the lowest variation among the plurality ofpotential 3D positions.
 9. The method of claim 8, wherein the pluralityof potential 3D positions of the CoR includes an initial value of thedistance from the center of an estimated corneal sphere.
 10. The methodof claim 9, wherein the initial value is an average of experimentallymeasured distance values.
 11. The method of claim 9, wherein the initialvalue is 4.7 mm.
 12. The method of claim 8, further comprisingdetermining an estimate of the corneal sphere based on analyzing theplurality of said eye images.
 13. The method of claim 8, wherein the CoRis the estimated position about which the eye rotates relative to atleast two axes.
 14. The method of claim 8, wherein obtaining theestimate of the 3D position of the CoR is performed during an initialuser calibration session of the display system.
 15. Non-transitorycomputer storage media storing instructions for execution by a displaysystem of one or more processors, wherein the display system isconfigured to project light to an eye of a user to display virtual imagecontent in a vision field of said user, said eye having a cornea, aniris, a pupil, a lens, a retina, and an optical axis extending throughsaid lens, pupil, and cornea, and wherein the instructions cause thedisplay system to: display virtual image content to the user's visionfield; receive a plurality of eye images of said eye of the user, theplurality of eye images captured by an inward-facing imaging systemconfigured to image said eye of the user to track movements of said eye;and obtain an estimate of a three-dimensional (3D) position of a centerof rotation (CoR) of said eye based on the plurality of said eye imagesobtained with said inward-facing imaging system, wherein obtaining theestimate of the 3D position of the CoR includes: determining a pluralityof potential 3D positions of the CoR, wherein each potential 3D positionof the CoR is determined as a particular distance from a center of anestimated corneal sphere that includes the cornea of the eye;calculating a variation for each of the potential 3D positions of theCoR, wherein the variation is calculated based on an analysis ofmultiple images of the user's eye captured over a period of time; anddetermining the estimate of the 3D position of the CoR as the potential3D position that exhibits the lowest variation among the plurality ofpotential 3D positions.
 16. The computer storage media of claim 15,wherein the plurality of potential 3D positions of the CoR includes aninitial value of the distance from the center of an estimated cornealsphere.
 17. The computer storage media of claim 16, wherein the initialvalue is an average of experimentally measured distance values.
 18. Thecomputer storage media of claim 16, wherein the initial value is 4.7 mm.19. The computer storage media of claim 15, wherein the instructionsfurther cause the display system to determine an estimate of the cornealsphere based on analyzing the plurality of said eye images.
 20. Thecomputer storage media of claim 15, wherein the CoR is the estimatedposition about which the eye rotates relative to at least two axes.